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DEPARTMENT  OF  THE  AIR  FORCE  AFOTEC  PAMPHLET  99^102,  VOLUME  2 

HQ  Air  Force  Operational  Test  and  Evaluation  Center  1  AUGUST  1994 

Test  and  Evaluation 

SOFTWARE  SUPPORT  LIFE  CYCLE  PROCESS  EVALUATION  GUIDE 


The  purpose  of  this  pamphlet  is  to  provide  the  Air  Force  Operational  Test  and  Evaluation  Center 
(AFOTEC)  software  test  manager  and  the  deputy  for  software  evaluation  information  needed  to 
evaluate  software  life  cycle  processes  as  they  influence  overall  software  supportability.  In  this 
pamphlet  are  the  means  to  track  the  processes  affecting  software  supportability,  beginning  as  early 
as  necessary  to  provide  insight  into  the  quality  of  the  evolving  software  products,  software  support 
resources,  and  operational  support  life  cycle  procedures  themselves. 

SUMMARY  OF  CHANGES 

AFOTEC  Pamphlet  (AFOTECPAM)  99-102  replaces  AFOTEC  Pamphlet  800-2,  all  volumes.  Minor 
administrative  changes  were  made  to  reflect  Air  Force  major  command  reorganizations.  This  volume 
is  the  second  in  a  series  of  Software  Operational  Test  and  Evaluation  pamphlets  prepared  by  the 
Software  Analysis  Team  at  Headquarters  (HQ)  AFOTEC.  Local  reproduction  of  all  volumes  in  this 
series  is  authorized.  Comments  should  be  directed  to  the  office  of  primary  responsibility.  The 
pamphlets  in  the  series  are: 

AFOTEC  Pamphlet  99-102,  volume  1  -  Management  of  Software  Operational  Test  and 

Evaluation 


AFOTEC  Pamphlet  99-102,  volume  2  -  Software  Support  Life  Cycle  Process  Evaluation  Guide 

AFOTEC  Pamphlet  99-102,  volume  3  -  Software  Maintainability  Evaluation  Guide 

AFOTEC  Pamphlet  99-102,  volume  4  -  Software  Usability  Evaluation  Guide 

AFOTEC  Pamphlet  99-102,  volume  5  -  Software  Support  Resources  Evaluation  Guide 

AFOTEC  Pamphlet  99-102,  volume  6  -  Software  Maturity  Assessment  Guide 

AFOTEC  Pamphlet  99-102,  volume  7  -  Software  Reliability  Evaluation  Guide 

AFOTEC  Pamphlet  99-102,  volume  8  -  Software  Operational  Assessment  Guide 

1.  General.  Software  supportability  is  a  measure  of  the  adequacy  of  products,  resources,  and 
procedures  to  facilitate  the  support  activities  in  establishing  an  operational  baseline,  modifying  and 
installing  software,  and  meeting  user  requirements.  Software  supportability  is  a  function  of  the 
quality  of  the  software  products,  the  capabilities  of  the  software  support  resources,  and  the  adequacy 
of  the  life  cycle  processes  that  affect  the  procurement,  development,  and  operational  support  of  the 
software.  The  focus  of  this  guide  is  the  life  cycle  processes  of  software  project  management  and  soft¬ 
ware  configuration  management  as  they  affect  the  eventual  supportability  of  fielded  software. 


2.  Overview: 

2.1.  You  should  read  pages  1  through  16  in  their  entirety  and  understand  the  evaluation  concept 
and  procedures  before  beginning  any  portion  of  an  evaluation.  These  pages  provide  you  with: 
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2.1.1.  A  background  of  the  AFOTEC  software 
support  life  cycle  process  evaluation  concept. 

2.1.2.  A  basic  understanding  of  the  evaluation 
procedures. 

2.1.3.  Detailed  instruction  for  using  the  soft¬ 
ware  life  cycle  process  questionnaires. 

2.2.  Attachment  1  contains  the  questionnaire 
and  explanatory  information  on  each  question. 
This  elaborating  information  is  provided  to 
ensure  that  you  fully  understand  the  intent  of 
each  question.  Included  are  definitions  of 
terms,  examples,  explanations,  and  special 
case  respotise  instructions,  as  necessary. 
Attachment  1  is  designed  to  be  used  as  the 
source  of  questions  for  the  evaluation.  At¬ 
tachment  1  is  also  designed  to  allow  you  to 
make  written  remarks  for  later  reference. 
Questions  are  not  listed  in  order  of  impor¬ 
tance. 

2.3.  Attachment  2  contains  a  summary  list  of 
all  the  questions  for  quick  reference.  Attach¬ 
ment  3  is  a  matrix  showing  when  the  ques¬ 
tions  should  be  answerable  by  milestone 
review  or  time  phase.  Attachment  4  is  a 
glossary  of  terms. 


3.  Methodology  Description  and  Factors: 

3.1.  Software  Life  Cycle  Process  Evalu¬ 
ation  Method: 

3.1.1.  The  method  for  evaluating  the  software 
life  cycle  process  is  based  on  the  use  of  closed- 
form  questions  with  optional  written 
comments  justifying  the  evaluation  score 
assigned  to  a  question.  This  questionnaire  is 
designed  to  determine  the  degree  to  which 
certain  desirable  attributes  or  characteristics 
affecting  software  supportability  are  or  will  be 
part  of  the  software  life  cycle  process.  The 
elements  of  the  software  life  cycle  process  and 
their  relationships  are  shown  in  figure  1  and 
are  described  in  the  following  paragraphs. 
The  hierarchical  evaluation  structure  shown 
in  the  figure  enables  you  to  identify  potential 
software  supportability  problems  at  various 
levels:  category/major  factor  (project  manage¬ 
ment,  configuration  management),  attributes 
or  characteristics  (planning,  organizational 
structure,  design  methods,  implementation 
methods,  test  strategies,  and  project 
interfaces),  low-level  characteristics  (individual 
questions),  or  some  combination.  Each  question 
should  be  evaluated  on  the  basis  of  the  attribute 
or  characteristic  to  which  it  is  attached. 
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Figure  1.  Software  Life  Cycle  Process  Evaluation  Hierarchy. 
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3.1.2.  Software  life  cycle  process  manage¬ 
ment  is  a  combination  of  the  policy,  method¬ 
ology,  procedures,  and  guidelines  applied  in  a 
software  environment  to  the  software  devel¬ 
opment  and  support  life  cycle  activities.  The 
major  management  aspects  for  purposes  of 
software  supportability  can  be  grouped  into 
two  major  factors:  software  project  manage¬ 
ment  and  software  configuration  manage¬ 
ment.  These  major  factor  characteristics  are 
evaluated  with  respect  to  their  impact  on 
software  supportability.  They  are  evaluated 
over  the  entire  life  cycle  of  a  system  with 
emphasis  on  three  activities:  procurement, 
development  contractor,  and  operational 
support.  Each  activity  concentrates  on  a 
series  of  events,  actions,  and  documents  that 
make  up  the  life  cycle  management  process. 

3.1.3.  Software  project  management  is  con¬ 
cerned  with  producing  a  software  product: 
either  the  initial  production  baseline  or  a 
version  of  the  production  baseline.  During 
development,  many  management  character¬ 
istics  will  influence  the  supportability  of  the 
software.  The  procurement  activity  will 
manage  the  overall  project  including  plan¬ 
ning  for  supportability.  The  development 
contractor  will  manage  the  delivery  of  the 
production  baseline  within  the  procurement 
activity  requirements.  During  the  post¬ 
deployment  of  a  system,  the  software  support 
activity  directly  controls  the  baseline  update 
process.  Lack  of  planning,  poor  organiza¬ 
tional  structure,  and  inadequate  design/ 
implementation/test  procedures  during  any 
activity  affect  the  supportability  of  the  result¬ 
ing  software  product. 

3.1.4.  Software  configuration  management 
will  provide  a  means  to: 

3. 1.4.1.  Identify  and  document  the  functional 
and  physical  characteristics  of  a  configura¬ 
tion  item. 

3. 1.4.2.  Control  changes  to  those  characteris¬ 
tics. 

3. 1.4.3.  Record  and  report  change  processing 
and  implementation  status. 

The  three  areas  that  produce  these  results 
are  configuration  identification,  configuration 
control,  and  configuration  status  accounting. 
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A  fourth  area,  configuration  audits,  verifies 
that  a  completed  product  and  its  documents 
meet  contractual  requirements.  The  procure¬ 
ment,  development,  and  operational  support 
activities  all  have  configuration  manage¬ 
ment  responsibilities  to  ensure  that  the 
baseline  production  products  and  subsequent 
revisions  are  properly  controlled.  These  con¬ 
figuration  management  responsibilities  have 
an  impact  on  the  supportability  of  the  soft¬ 
ware  products. 

3.2.  Software  Configuration  Manage¬ 
ment  Evaluation  Factors.  The  software 
project  management  evaluation  is  based  on 
six  characteristics  or  evaluation  factors: 
planning,  organization  structure,  design 
methods,  implementation  methods,  test  stra¬ 
tegies,  and  project  interfaces.  The  following 
paragraphs  define  these  factors  and  discuss 
their  application  in  the  evaluation  process. 

3.2.1.  Planning: 

3.2. 1.1.  Planning  is  evaluated  for  how  well 
the  life  cycle  plans  address  software  support- 
ability.  Software  project  management  plan¬ 
ning  enhances  software  supportability  to  the 
extent  that  plans  for  the  development,  test, 
product  transfer,  and  operational  support 
exist  have  been  implemented,  have  been 
appropriately  coordinated  across  responsible 
agencies,  and  satisfy  contractual  and/or  regu¬ 
lation  requirements. 

3.2. 1.2.  Major  planning  documents  for  the 
procurement  activity  include  the  program 
management  directive  (PMD),  program  man¬ 
agement  plan  (PMP),  test  and  evaluation 
master  plan  (TEMP),  computer  resources 
lifecycle  management  plan  (CRLCMP),  de¬ 
velopment  test  and  evaluation  (DT&E)  plans, 
and  operational  test  and  evaluation  (OT&E) 
plans. 

3.2. 1.3.  Major  planning  documents  for  the 
development  contractor  activity  include  the 
statement  of  work  (SOW),  system/segment 
specification  or  prime  item  development 
(PID)  specification,  software  development 
plan  (SDP),  software  configuration  manage¬ 
ment  plan  (SCMP),  software  quality  program 
plan  (SQPP),  software  standards  and  pro¬ 
cedures  manual  (SSPM),  the  software  test 
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plan  (STP),  and  the  computer  resources 
integrated  support  document  (CRISD). 

3.2. 1.4.  Major  planning  documents  for  the 
operational  support  activity  include  the 
TEMP,  DT&E  and  OT&E  plans,  and 
CRLCMP. 

3.2. 1.5.  One  of  the  most  important  results  of 
good  planning  is  the  coordination  of  informa¬ 
tion  across  the  various  planning  documents 
to  minimize  redundancy  and  satisfy  the  nec¬ 
essary  content  requirements  of  the  plans. 
The  conciseness  and  level  of  detail  of  plan¬ 
ning  information  is  very  important.  Fre¬ 
quently,  plans  act  as  a  place  holder  for  "real" 
information,  serving  a  role  only  a  little  better 
than  a  "TBD."  To  say  that  structured  pro¬ 
gramming  standards  will  be  followed  is  not 
precise  enough  in  the  SSPM.  Precise  pro¬ 
gramming  requirements  which  represent  the 
contractor's  definition  of  "structured  pro¬ 
gramming  standards"  must  be  specified  in  a 
manner  suitable  for  quality  assurance  testing 
for  conformance.  As  another  example,  it  is 
not  satisfactory  to  indicate  in  the  CRLCMP 
that  the  support  resources  space  require¬ 
ments  are  4,800  square  feet.  It  is  necessary 
to  indicate  how  that  space  is  allocated  among 
support  personnel  office  space,  support 
hardware  space,  storage/library  space,  and 
any  other  space  allocations  which  might  be 
peculiar  to  the  application.  Furthermore,  a 
top-level  facility  layout  showing  the  physical 
relationship  among  the  space  allocations  is 
appropriate. 

3.2. 1.6.  When  systems  have  interservice 
operability  requirements,  plans  for  the  ap¬ 
propriate  interservice  interfaces  and  joint 
activities  should  be  clearly  specified,  particu¬ 
larly  the  plans  for  supporting  the  software. 

3.2. 1.7.  When  development  contractor  activ¬ 
ity  involves  subcontractors,  the  plans  for 
managing  the  subcontractor  effort  and  the 
subcontractor  internal  plans  for  managing 
their  efforts  should  be  clearly  specified,  par¬ 
ticularly  the  plans  for  supporting  deliverable 
software. 

3.2. 1.8.  A  good  plan  will  possess  a  concise 
statement  of  the  objectives  of  the  plan,  the 
techniques  and  methods  by  which  the  plan 
will  be  implemented,  the  responsible  organ¬ 


izational  elements  for  making  sure  the  plan 
is  implemented  and  evolved  as  necessary,  the 
schedule  by  which  the  objectives  of  the  plan 
are  to  be  accomplished,  and  the  relationships 
of  the  plan  to  any  other  system  elements. 

3.2.2.  Organizational  Structure: 

3.2.2. 1.  The  software  project  management 
organizational  structure  enhances  software 
supportability  to  the  extent  that  the  physical 
structure,  functional  responsibilities,  exter¬ 
nal  interfaces,  and  assigned  personnel  pro¬ 
vide  for  continuity  over  the  software  life  cycle 
phases  and  have  proper  interfaces  with 
organizations  responsible  for  software  sup- 
portability. 

3. 2.2.2.  The  procurement  activity  must  have 
an  organizational  structure  which  provides 
continuity  across  all  life  cycle  phases  and 
through  each  milestone.  The  organization 
structure  must  provide  for  adequate  dissemi¬ 
nation  and  coordination  of  information 
among  all  activities.  Organization  elements 
must  provide  functions  for  project  oversight 
(plans  and  policies),  configuration  manage¬ 
ment,  quality  evaluation,  project  reviews  and 
audits,  testing  and  evaluation,  and  transfer 
of  responsibility. 

3. 2.2.3.  The  development  contractor  activity 
must  have  an  organizational  structure  which 
matches  the  work  breakdown  structure  and 
provides  continuity  throughout  all  engineer¬ 
ing  and  manufacture  development  activities 
and  the  transition  into  postdeployment  sup¬ 
port.  Appropriate  organizational  elements 
should  exist  for  internal  configuration  man¬ 
agement,  quality  assurance,  test  and  evalua¬ 
tion,  product  development,  and  procurement/ 
support  contractual  interface  activity. 

3. 2.2.4.  The  operational  support  activity 
must  have  an  organizational  structure  which 
satisfies  mission  requirements  within  the 
requirements  imposed  by  the  procurement 
and  development  activity  organization  and 
applicable  regulations  and  directives. 
Organization  elements  should  be  established 
early  in  the  development  phase  to  ensure 
proper  transition  to  postdeployment  support 
through  understanding  of  the  software 
support  requirements. 
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3.2.3.  Design  Methods: 

3.2.3. 1.  Design  methods  are  evaluated  for 
characteristics  which  indicate  that  software 
supportability  has  been  designed  into  the 
software  products.  Software  project  man¬ 
agement  utilizes  design  methods  which  en¬ 
hance  software  supportability  to  the  extent 
that  design  methodology  standards  and  con¬ 
ventions  (1)  are  documented,  followed,  and 
validated  through  quality  assurance;  (2)  can 
be  transitioned  to  support  activity;  and  (3) 
produce  adequate  design  specifications  which 
reflect  supportability  characteristics. 

3.2.3.2.  The  procurement  activity  design 
methods  are  reflected  in  the  requirements 
imposed  upon  the  development  contractor 
activity  through  the  system/segment  specifi¬ 
cation  and  the  request  for  proposal.  Pro¬ 
curement  monitoring  of  development  con¬ 
tractor  design  activities  and  acceptance  of 
those  activities  are  also  a  reflection  of  the 
procurement  activity  design  methods. 

3.2. 3.3.  The  development  contractor  activity 
design  methods  should  be  defined  in  an  inter¬ 
nal  standards  and  convention  manual  and 
validated  by  a  quality  assurance  function. 

The  methods  should  reflect  use  of  a 
consistent  methodology,  traceability  between 
requirements  and  production  products, 
traceability  of  design  decisions,  use  of 
abstraction  and  information  hiding,  and  use 
of  techniques  to  enhance  the  software 
product  characteristics  of  modularity, 
descriptiveness,  consistency,  simplicity,  ex¬ 
pandability,  and  instrumentation.  Auto¬ 
mated  tool  support  as  an  aid  to  development 
design  and  support  design  evolution  is  an 
important  part  of  the  development  contractor 
design  methods. 

3.2. 3.4.  The  operational  support  activity 
design  methods  should  be  defined  at  a  high 
level  in  a  procurement  activity  requirements 
specification  and  at  a  lower  level  by  an  inter¬ 
nal  support  standards  and  conventions 
manual.  The  methods  should  have  a  close 
similarity  to  the  methods  used  by  the  devel¬ 
opment  contractor  activity  in  order  to  facili¬ 
tate  transition  of  the  software  design  evolu¬ 
tion  to  the  support  activity. 

3.2.4.  Implementation  Methods: 
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3. 2.4.1.  The  software  project  management 
process  uses  implementation  methods  which 
enhance  software  supportability  to  the  extent 
that  implementation/coding/testing  method¬ 
ology,  standards,  and  conventions  (1)  are 
documented,  followed,  and  validated  through 
quality  assurance;  (2)  can  be  transitioned  to 
the  support  activity;  and  (3)  produce  sup¬ 
portable  production  products. 

3.2.4.2.  The  procurement  activity  implemen¬ 
tation  methods  are  reflected  in  the  require¬ 
ments  (contract  specifications)  imposed  on 
the  development  contractor  activity  for 
implementation  and  coding  standards  and 
the  process  through  which  such  standards 
and  the  form  of  the  production  products  are 
reviewed  and  accepted  for  operational  use, 

3.2.4.3.  The  development  contractor  activity 
implementation  methods  should  be  defined  in 
an  internal  standards  and  conventions 
manual  and  validated  by  a  quality  assurance 
function.  The  methods  should  reflect  use  of 
acceptable  implementation  team  organiza¬ 
tional  strategies.  For  example,  the  chief 
programming  team  methods  should  enhance 
traceability  among  requirements,  design,  and 
product.  The  methods  should  emphasize 
techniques  to  enhance  the  software  product 
characteristics  of  modularity,  descriptiveess, 
consistency,  simplicity,  expandability,  and 
testability.  Automated  tool  support  to  aid  de¬ 
velopment  implementation  and  change  pro¬ 
cessing  is  an  important  part  of  development 
contractor  implementation  methods. 

3. 2. 4. 4.  The  operational  support  activity 
implementation  methods  should  be  defined  at 
a  high  level  in  a  procurement  activity 
requirements  specification  and  other  support 
documents  such  as  the  CRLCMP.  Specific 
methods  should  be  defined  at  a  low  level  by 
an  internal  standards  and  conventions 
manual.  The  methods  should  have  a  close 
similarity  to  the  methods  used  by  the 
development  contractor  activity  in  order  to 
facilitate  transition  of  the  software 
implementation  evolution  to  the  support 
activity. 

3. 2. 4. 5.  Implementation  methods  are  evalu¬ 
ated  for  consistency  with  standards,  avail¬ 
ability  of  automated  tool  support  capabilities 
in  the  form  of  software  benches  and  inte- 
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grated  laboratory  testbeds,  potential  for 
effective  use  during  software  support,  and 
the  traceability  of  implemented  product 
status  and  top-level  requirements. 

3.2.5.  Test  Strategies: 

3.2.5. 1.  Test  strategies  are  evaluated  for  how 
well  the  strategies  provide  for  a  delivery  of 
mature  software  products  and  retest  of  those 
products  during  software  support  activities. 
Software  project  management  utilizes  test 
strategies  which  enhance  software  support- 
ability  to  the  extent  that  the  test  plans, 
descriptions,  procedures,  and  results  (1)  have 
been  documented,  (2)  can  be  transitioned  to 
and  reused  by  the  support  activity,  and  (3) 
provide  for  a  consistent  and  systematic  proc¬ 
ess  for  verifying  and  validating  that  software 
requirements  have  been  satisfied. 

3. 2. 5.2.  The  procurement  activity  test  strate¬ 
gies  are  documented  in  the  TEMP,  DT&E 
plans  and  reports,  OT&E  plans  and  reports, 
optionally  in  independent  verification  and 
validation  (IV&V)  plans  and  reports,  the 
preliminary  and  formal  qualification  tests 
(preliminary  qualification  tests  (PQT)  and 
formal  qualification  tests  (FQT)),  and  the 
acceptance  strategies  which  revolve  around 
formal  reviews  (e.g.,  system  requirements 
review  (SRR),  preliminary  design  review 
(PDR),  critical  design  review  (CDR),  test 
readiness  review  (TRR))  and  audits  (e.g., 
functional  configuration  audit  (EGA), 
physical  configuration  audit  (PC A)).  The  test 
strategies  should  clearly  indicate  software 
test  objectives,  relationships  to  system  test 
objectives,  relationships  among  the  various 
test  organizations  (e.g.,  DT&E,  OT&E, 
IV&V),  contractually  binding  aspects  of  tests 
such  as  the  schedule  and  deliverables,  and 
precisely  what  tests  will  be  required  prior  to 
acceptance  of  the  production  product.  Test 
strategies  should  describe  how  software  test 
discrepancies  will  be  documented  and 
tracked,  corrections  coordinated,  and  results 
passed  to  the  test  and  eventual  support 
organizations.  A  test  strategy  for  the 
transition  period  after  the  production 
decision  should  be  specifically  addressed. 

3.2. 5.3.  The  development  contractor  activity 
test  strategies  are  documented  in  test  plans, 
procedures,  and  reports.  Automated  support 


in  the  form  of  software  benches  (individual 
module  tests),  laboratory  integrated  testbeds, 
and  operational  integrated  systems  have  a 
major  impact  on  the  effectiveness  of  the  tests. 
A  clear  strategy  for  use  of  such  tools  should 
be  documented,  used,  and  transitioned  early 
to  the  support  activity.  These  test  strategies 
should  address  (1)  features  to  be  tested,  (2) 
traceability  to  the  requirements  specifica¬ 
tions,  and  (3)  among  the  various  test  docu¬ 
ments,  a  consistent  approach  to  testing 
various  levels  (e.g.,  unit,  integrated,  sys¬ 
tem),  environmental  requirements,  organiza¬ 
tional  responsibilities  and  interfaces,  sched¬ 
ule,  deliverables,  risk  and  contingencies,  and 
acceptance/approval  criteria. 

3.2.5.4.  The  operational  support  activity  test 
strategies  documentation  is  similar  to  the 
procurement  activity  (e.g.,  for  the  TEMP  and 
FOT&E  plans  as  dynamic  documents)  and 
the  development  contractor  activity  (e.g.,  via 
transition  of  the  test  plans/procedures/ 
results  and  automated  tools  to  the  support 
activity).  Coordination  between  the  opera¬ 
tional  activity  and  support  activity  test 
strategies  is  important  during  postdeploy¬ 
ment  because  of  the  requirement  to  use 
operational  testbeds.  This  coordination 
should  be  reflected  via  resource  requirements 
in  top-level  planning  documents  such  as  the 
TEMP,  CRLCMP,  and  specific  software  sup¬ 
port  management  project  (i.e.,  block  release) 
internal  documents.  Similar  test  strategy 
characteristics  as  in  subparagraphs  3. 2. 5. 2 
and  3. 2. 5.3  above  should  be  present  in  the 
operational  support  activity  test  strategies. 

3.2.6.  Project  Interfaces: 

3.2.6. 1.  Organizational  interfaces  are  evalu¬ 
ated  for  their  effectiveness  in  resolving 
interorganization  issues  concerning  software 
support.  Software  project  management  pos¬ 
sesses  organizational  interfaces  which  en¬ 
hance  software  supportability  to  the  extent 
that  external  project  organization  relation¬ 
ships  and  responsibilities  (1)  are  defined,  (2) 
provide  a  valuable  functional  role,  and  (3) 
contribute  to  systematic  cost  effective  pro¬ 
curement,  development,  operation,  and  sup¬ 
port  processes. 

3.2.6.2.  The  procurement  activity  organiza¬ 
tional  interfaces  are  primarily  with  the  devel- 
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opment  contractor  activity,  the  operational 
support  activity,  interface  working  groups 
required  by  regulations,  and  other  higher 
level  groups  (e.g.,  military,  Department  of 
Defense  (DoD),  federal  government  agencies. 
Congress,  public).  An  IV&V  interface  may 
also  be  required.  It  is  necessary  that  each  of 
these  interfaces  be  defined  to  a  level  of  detail 
consistent  with  the  particular  application 
system.  Function  and  responsible  persons 
should  be  identified. 

3.2.6.3.  The  development  contractor  activity 
organizational  interfaces  include  the  procure¬ 
ment  activity,  interface  working  groups,  and 
higher  level  internal  organization  elements 
(e.g.,  corporate  management).  An  IV&V 
interface  may  also  be  required.  In  addition, 
if  subcontractors  are  involved,  this  interface 
must  be  clearly  established.  Function  and 
schedule  of  contact  should  be  defined  and 
responsible  persons  identified. 

3.2.6.4.  The  operational  support  activity 
organizational  interfaces  are  very  similar  to 
those  of  the  procurement  activity. 

3.3.  Software  Configuration  Manage¬ 
ment  Evaluation  Factors.  The  software 
configuration  management  evaluation  is 
based  on  four  characteristics  or  test  factors: 
software  configuration  identification,  soft¬ 
ware  configuration  control,  software  status 
accounting,  and  software  audits.  Definitions 
of  these  test  factors  and  discussion  of  their 
application  in  the  evaluation  process  are 
given  in  the  following  paragraphs. 

3.3.1.  Software  Configuration  Identifica¬ 
tion: 

3.3. 1.1.  Configuration  identification  is  evalu¬ 
ated  for  how  well  the  controlled  baselines  are 
identified,  unique  identification  problems 
such  as  multiple  locations/version  variations 
are  solved,  compliance  with  regulations  and 
standards,  and  the  use  of  automated  tools  to 
support  generation  and  update  of  configura¬ 
tion  indexes  for  the  baselines.  Software  con¬ 
figuration  management  uses  configuration 
identification  to  enhance  software  support- 
ability  to  the  extent  that  the  software  docu¬ 
mentation  properly  identifies  the  configura¬ 
tion  items,  their  characteristics,  and  their 
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relationships  according  to  required  standards 
and  regulations. 

3.3. 1.2.  The  procurement  activity  is  respon¬ 
sible  for  following  existing  guidelines  and 
regulations  for  identification  of  software  con¬ 
figuration  items  and  ensuring  that  these 
guidelines  and  regulations  are  contractually 
required  by  the  development  contractor.  The 
procurement  activity  is  also  responsible  for 
monitoring  contractor  use  of  the  guidelines 
and  regulations  to  ensure  that  the  functional, 
allocated,  developmental,  and  production 
software  baselines  are  properly  identified. 

3.3. 1.3.  The  development  contractor  activity 
is  responsible  for  following  contractual  re¬ 
quirements  for  configuration  management. 
This  should  include  development  of  a 
software  configuration  management  plan  in 
which  configuration  identification  standards 
and  procedures  for  the  controlled  software 
baselines  are  specified.  Independent  of  con¬ 
tractual  requirements,  internal  configuration 
identification  standards  and  procedures 
should  exist. 

3. 3. 1.4.  The  operational  support  activity  is 
responsible  for  continuation  of  the  same  con¬ 
figuration  identification  requirements  as 
required  for  the  development  contractor 
activity.  In  addition,  certain  monitoring 
responsibilities  of  the  procurement  activity 
are  ensured  by  the  operation  support 
activity.  The  CRLCMP  is  the  primary  op¬ 
eration  support  activity  software  configura¬ 
tion  management  planning  document. 

3.3.2.  Software  Configuration  Control: 

3.3.2. 1.  Configuration  control  is  evaluated 
for  how  well  changes  to  the  functional, 
allocated,  developmental,  and  production 
baselines  are  controlled.  This  evaluation 
includes  the  adequacy  of  control  procedures 
and  forms,  the  capability  to  transition  such 
procedures  to  the  support  activity  proce¬ 
dures,  the  adequacy  of  the  interface  control 
among  the  organizations  responsible  for  some 
aspect  of  configuration  control,  and  the  use  of 
automated  tools  to  protect  inadvertent 
change  and  assist  in  administering  approved 
changes.  Software  configuration  manage¬ 
ment  uses  configuration  control  to  enhance 
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software  supportability  to  the  extent  change 
decisions  to  software  baselines  are  made, 
administered,  and  implemented  in  a  con¬ 
trolled  environment. 

3. 3.2.2.  The  procurement  activity  makes 
decisions  regarding  changes  to  the  develop¬ 
mental  software  baselines  through  a  system 
configuration  control  board.  Any  change  re¬ 
quest  to  a  functional,  allocated,  or  production 
software  baseline  must  be  approved  by  the 
procurement  configuration  control  board. 
The  changes  are  administered  by  the 
program  office's  configuration  management 
organization.  Implementation  is  generally 
accomplished  by  the  development  contractor 
activity. 

3. 3.2. 3.  The  development  contractor  activity 
makes  decisions  regarding  changes  to  soft¬ 
ware  product  baselines  (prior  to  the  pro¬ 
curement  agency  taking  formal  control  of  the 
product  baseline).  Administration  of  such 
changes  should  be  through  an  internal  con¬ 
figuration  management  organization.  Im¬ 
plementation  is  by  project  software  person¬ 
nel.  Direction  for  changes  to  the  functional 
and  allocated  baselines  continue  to  come 
from  the  procurement  agency.  In  addition, 
changes  to  the  functional,  allocated,  or 
production  baseline  (prior  to  operational 
support)  are  implemented  by  the  develop¬ 
ment  contractor  activity.  Interfaces  among 
participating  contractors  must  be  established 
to  maintain  proper  configuration  control  of 
the  developmental  products. 

3. 3.2.4.  The  operational  support  activity 
assumes  the  responsibility  to  implement  any 
change  requests  from  the  development 
contractor.  Frequently,  some  level  of 
configuration  control  is  accomplished  by  the 
support  activity  prior  to  that  to  ease  the 
transition.  Decisions  for  making  changes 
once  in  operational  use  are  shared  among  the 
using  command,  supporting  command/ 
activities,  any  interservice  commands  as 
appropriate,  and  support  contractors  as 
appropriate.  The  CRLCMP  is  the  primary 
planning  document  for  the  operational  sup¬ 
port  activity  software  configuration  manage¬ 
ment.  In  addition,  using/supporting  com¬ 
mand/activities  and  subordinate  agency  reg¬ 
ulations  may  exist. 


3.3.3.  Software  Status  Accounting: 

3.3.3. 1.  Status  accounting  is  evaluated  for 
how  well  the  changes  to  software  baselines 
are  tracked  and  reported,  the  capability  of 
automated  tools  to  support  the  tracking,  and 
the  effectiveness  of  interfaces  for  communi¬ 
cating  status  accounting  information  among 
organizational  elements.  Software  configura¬ 
tion  management  uses  status  accounting  to 
enhance  software  supportability  to  the  extent 
that  configuration  identification  and  changes 
to  the  configured  items  are  tracked  and  re¬ 
ported  through  a  configuration  index  and 
change  status  reports. 

3.3.3. 2.  The  procurement  activity  is  respon¬ 
sible  for  monitoring  the  status  of  the  baseline 
development.  Status  accounting  provides  the 
procurement  activity  with  visibility  and 
traceability  of  baseline  configurations  and 
their  changes.  The  program  office  configu¬ 
ration  management  organization  uses  status 
accounting  reports  to  maintain  official  base¬ 
lines  and  to  perform  the  system  configuration 
control  board  function. 

3. 3. 3.3.  The  development  contractor  activity 
uses  status  accounting  information  (configu¬ 
ration  index  and  change  reports)  for  internal 
management  visibility  and  traceability  and 
or  external  government  reporting  require¬ 
ments. 

3. 3.3.4.  The  operational  support  activity  uses 
status  accounting  information  for  coordina¬ 
tion  of  software  maintenance  tasks  that  may 
involve  many  organizations  in  widely  scat¬ 
tered  locations  as  well  as  for  usual  internal 
management  visibility  and  implementation 
change  status.  The  CRLCMP  is  the  primary 
operational  support  software  configuration 
management  planning  document. 

3.3.4.  Software  Configuration  Audit/Re¬ 
view: 

3. 3.4.1.  Software  configuration  audit/review 
is  evaluated  for  adherence  to  regulations  and 
standards  (such  as  MIL-STD-1521B)  and  for 
the  planning/conduct/results  associated  with 
such  audits/reviews.  Software  configuration 
management  utilizes  configuration  audits/re¬ 
views  to  enhance  software  supportability  to 
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the  extent  that  the  functional  and  physical 
configuration  of  the  software  baselines  have 
contract  requirements. 

3.3.4.2.  The  procurement  activity  is  respon¬ 
sible  for  preparation  and  approval  of  the 
formal  audits  and  reviews:  functional  con¬ 
figuration  audit,  physical  configuration  audit, 
and  formal  qualification  review. 

3.3.4.3.  The  development  contractor  activity 
is  responsible  for  preparation  of  and  con¬ 
ducting  or  assisting  with  formal  configura¬ 
tion  audits  and  reviews.  In  addition,  internal 
configuration  audits  should  be  periodically 
done  on  developmental  baselines  to  provide 
assurance  that  configuration  identification, 
control,  and  status  accounting  functions  are 
being  properly  administered  and  the  result¬ 
ing  configuration  information  is  consistent. 

3.3.4.4.  The  operational  support  activity  is 
responsible  for  monitoring  the  formal  audits 
and  reviews  prior  to  operational  support  and 


for  preparation  and  conduct  of  updated  base¬ 
line  configuration  audits  and  reviews  after 
operational  support.  The  CRLCMP  is  the 
primary  operational  support  activity  software 
configuration  management  planning  document. 

4.  Software  Support  Life  Cycle  Process 
Evaluation  Procedure.  The  software  sup¬ 
port  life  cycle  process  evaluation  procedure  is 
an  ongoing  effort  throughout  AFOTEC's 
involvement  with  a  system  containing  mis¬ 
sion  critical  computer  resources.  The  par¬ 
ticular  life  cycle  process  evaluation  emphasis 
by  activity  and  life  cycle  phase  is  illustrated 
in  figure  2.  The  numbers  in  parentheses  are 
the  relative  weights  of  emphasis.  The  spe¬ 
cific  aspects  of  the  evaluation  are  briefly 
described  in  the  following  paragraphs. 

4.1.  Planning  the  Evaluation: 

4.1.1.  You  must  carefully  plan  for  the  collec¬ 
tion  of  the  required  data  in  order  to 
adequately  complete  the  life  cycle  process 
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Figure  2.  Focus  of  Software  Life  Cycle  Process  by  Activity. 
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evaluation  questionnaire.  You  should  review 
the  software  support  life  cycle  process 
questionnaire  during  AFOTEC's  advanced 
planning  and  identify  likely  sources  for 
answers  to  the  questions.  A  timetable  should 
be  developed  as  part  of  the  evaluation  plan 
which  specifies  when  the  identified  source 
documents  will  be  available,  program  reviews 
will  be  held,  tests  will  be  conducted,  and  key 
personnel  can  be  visited  to  obtain  the 
information  needed  to  address  each  question. 

4.1.2.  Most  sources  of  information  will  be 
similar  between  systems.  For  the  most  part, 
this  evaluation  guide  can  serve  as  a  checklist 
(from  advanced  planning  through  OT&E)  for 
AFOTEC  concerns  about  the  software  life 
cycle  processes  that  ultimately  impact  soft¬ 
ware  supportability  once  a  system  is  fielded. 

4.2.  Conducting  the  Evaluation: 

4.2.1,  Conducting  the  software  life  cycle 
process  evaluation  consists  of  the  formal 
completion  of  the  questionnaire  as  you  track  a 
system's  development  and  plan  for  and 
conduct  the  OT&E,  Space  is  provided  in  each 
question  for  both  the  response  score  and 
response  rationale.  Use  the  rationale  portion 
to  provide  comments  and  sources  of  the 
information  justifying  the  response.  This  will 
assist  in  (1)  writing  the  report  and  (2) 
transitioning  the  program  to  another  software 
test  manager  or  deputy  for  software 
evaluation.  The  intent  is  not  to  try  to  complete 
this  evaluation  in  one  sitting  but  to  work 
through  it  during  your  involvement  with  a 
program.  If  you  are  starting  out  on  a  new 
program,  this  evaluation  guide  can  be  used 
directly  as  you  begin  advanced  planning.  Use 
the  questions  for  meeting  preparation,  as  they 
provide  a  good  source  of  topics  that  need  to  be 
discussed,  and  document  review.  If  you  are  in 
the  middle  of  a  program,  previously  collected 
data  or  historical  searches,  updated  to  reflect 
current  software  life  cycle  process  status,  can 
be  used  as  a  basis  for  a  particular  response. 

4.2.2  Some  questions  may  not  be  answerable 
in  a  direct  manner.  In  this  case,  you  will  have 
to  use  your  best  judgment  estimate  or  leave  it 
unanswered.  You  will  not  invalidate  the 
evaluation,  but  the  basis  of  the  upper-level 
characteristic  or  test  factor  to  which  the 
unanswered  question  pertains  has  a  reduced 


confidence.  If  an  early  operational  evaluation 
is  being  conducted,  some  areas  may  have  to  be 
addressed  at  the  upper-level  characteristic  or 
test  factor.  Again,  this  does  not  constitute  an 
invalid  evaluation,  but  care  must  be  taken  in 
reporting  the  findings  since  the  confidence 
level  might  be  quite  low. 

4.3.  Analyzing  Evaluation  Results: 

4.3.1.  Problems/concerns  that  you  note  during 
any  phase  of  your  involvement  with  a  program 
can  be  presented  in  an  appropriate  forum 
(such  as  a  system/software  requirements 
review,  preliminary  design  review,  engineer¬ 
ing  design  review,  or  program  management 
review)  or  staffed  through  HQ  AFOTEC  "T"  or 
Systems  Analysis  Directorates  to  the  imple¬ 
menting  agency.  Lost  opportunities  to 
address  concerns/problems  (or  unresolved  con¬ 
cerns/problems)  will  simply  remain  as 
evaluated.  Areas  that  are  resolved  favorably 
with  regard  to  the  evaluation  questions  must 
result  in  reevaluating  that  question. 

4.3.2.  Processing  the  numerical  scores  from 
each  question  into  its  upper-level  charac¬ 
teristic  or  test  factor  can  be  accomplished  in 
two  ways:  by  inputting  the  question  scores 
directly  into  the  automated  Field  Question 
Analysis  System  (FQAS)  or  by  marking 
National  Computer  System  (NCS)  general 
purpose  answer  sheets  (described  below)  for 
computing  the  characteristic  or  test  factor 
averages.  Copies  of  FQAS  are  available  from 
AFOTEC/SAS. 

5.  Response  Form: 

5.1.1.  The  name  block  is  shown  in  figure  3. 
Only  your  name  needs  to  be  put  here.  The 
SEX  and  GRADE  or  EDUC  blocks  are  not 
used. 

5.1.2.  The  numerical  identification  block  is 
shown  in  figure  4.  Each  assigned  column  has 
special  meaning,  so  use  extreme  care  in 
entering  the  data.  Integer  values  represent 
specific  characteristics  or  test  factors  as  shown 
in  figure  4. 

5.1.3.  The  question  response  block  is  shown  in 
figure  5.  The  response  number  corresponds  to 
the  question  number  for  the  characteristic  or 
test  factor  being  addressed. 


AFOTECPAM  99-102  Volume  2  1  June  1994 


11 


(Note:  The  "SEX”  block  and  "GRADE"  or  "EDUC"  blocks  need  not  be  filled  in.) 


Figure  3.  Evaluator  Name  Block  Example. 

5.2.  This  form  is  processed  through  an  optical 
scanner,  so  it  is  important  that  the 
appropriate  circles  be  darkly  marked  and  no 
extraneous  marks  appear.  Errors  must  be 
completely  erased! 

6.  Response  Scale: 

6.1.  To  complete  the  evaluation  questionnaire 
within  this  volume,  you  will  use  the  subjective 
scale  of  agreement  from  6  (completely  agree) 
to  1  (completely  disagree).  In  general,  the 
response  scale  should  be  interpreted  as 
follows: 


(A)  COMPLETELY  AGREE  (6):  There  must 
be  absolutely  no  doubt  when  using  this  re¬ 
sponse  that  the  characteristic  being  evaluated 
is  totally  satisfactory  with  respect  to  the 
characteristic  addressed. 

(B)  STRONGLY  AGREE  (5):  This  response 
indicates  that  the  characteristic  being 
evaluated  is  very  good  and  is  very  helpful  for 
software  supportability. 

(C)  GENERALLY  AGREE  (4):  This  response 
indicates  that  the  characteristic  being  evalua¬ 
ted  is  satisfactory  but  may  require  improve- 
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Columns 

Description 

Range 

Birthdate 

Date  Evaluation  Started 

-  Month  (MO) 

Jan  -  Dec 

-  Day 

01-31 

-  Year(YR) 

Identification  Number 

00-99 

-  A 

Major  Characteristic 

5  =  Project  Management 

6  =  Configuration  Management 

-  B 

Lower  Level  Characteristic 

1  (PM  -  Planning,  CM  -  Identification) 

2  (PM  -  Org  Structure,  CM  -  Control) 

3  (PM  -  Design  Methods,  CM  -  Status  Accounting) 

4  (PM  -  Implementation  Methods,  CM  -  Review/Audit) 

5  (PM  -  Test  Strategies) 

6  (PM  -  Project  Interfaces) 

-  C,D 

System  Level  Code  (Level  1) 

01-99 

-  E,F 

Subsystem  Code  (Level  2) 

01-99 

-  G,H,I 

Special  Codes  (Leave  Blank) 

Evaluator  Code 

001-999  (Use  only  if  more  than  one 
person  is  performing  the  evaluation) 

Figure  4.  Numerical  Identification  Block  Format  and  Example. 


merits  to  make  it  helpful  for  software  sup- 
portability. 

(D)  GENERALLY  DISAGREE  (3):  This  re¬ 
sponse  indicates  that  the  characteristic  being 
evaluated  is  unsatisfactory  and  some  im¬ 
provements  are  required  to  make  it  helpful  for 
software  supportability. 

(E)  STRONGLY  DISAGREE  (2):  This  re¬ 
sponse  indicates  that  the  characteristic  being 
evaluated  is  unsatisfactory  and  major  im¬ 
provements  are  required  before  it  would  be 
helpful  for  software  supportability. 

(F)  COMPLETELY  DISAGREE  (1):  There 
must  be  absolutely  no  doubt  when  using  this 


response  that  the  characteristic  being  eval¬ 
uated  is  totally  unsatisfactory  with  respect  to 
the  characteristic  addressed. 

(H)  NOT  ANSWERED  (n/a):  This  response 
indicates  that  the  characteristic  being  evalu¬ 
ated  is  not  answerable  at  this  time  or  is  being 
deliberately  bypassed. 

One  of  these  responses  should  be  given  for 
each  question.  Also,  response  1  or  6  is,  in 
general,  not  expected,  since  these  responses 
indicate  a  worst  possible  or  best  possible 
characteristic  relative  to  software  file  cycle 
processes  in  general. 
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RESPONSE  SCALE 

6  =  A  2  =  E 

5  =  B  1  =  F 

4  =  C  n/a=  H 

3  =  D 


Figure  5.  Question  Response  Forward  Block. 
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Directives,  Regulations,  Standards 

1.  DoDD  5000.1,  Major  System  Acquisition,  23  February  1991. 

2.  DoDD  5000.2,  Major  System  Acquisition  Procedures,  23  February  1991. 

3.  DoDD  3405.1,  Computer  Programming  Language  Policy,  2  April  1987. 

4.  AFR  55-43,  Management  of  Operational  Test  and  Evaluation,  28  June  1985  (to  be  published  as 
AFI  99-102). 

5.  AFR  80-14,  Test  and  Evaluation,  3  November  1986  (to  be  published  as  AFI  99-101). 

6.  AFR  800-8,  Integrated  Logistics  Support  (ILS)  Program,  25  June  1986. 

7.  AFR  800-14,  Lifecycle  Management  of  Computer  Resources  in  Systems,  29  September  1986  (in 
revision). 

8.  AFP  800-48,  Software  Management  Indicators,  June  1992. 

9.  DoD-STD-2167A,  Defense  System  Software  Development,  29  February  1988,  and  associated 
Data  Item  Descriptions. 

10.  DoD-STD-2168,  Software  Quality  Evaluation,  29  April  1988,  and  associated  Data  Item  Descrip¬ 
tion. 

11.  DoD-HDBK-287M  Defense  System  Software  Development  Handbook,  23  May  1986. 

12.  DoD-STD-480A,  Configuration  Control  -  Engineering  Changes,  Deviations,  and  Waivers,  12 
April  1978. 

13.  MIL-STD-482A,  Configuration  Status  Accounting  Data  Elements  and  Related  Features,  1  April 
1974. 

14.  MIL-STD-483A,  Configuration  Management  Practices  for  Systems,  Equipment,  Munitions,  and 
Computer  Programs,  4  June  1985. 

15.  MIL-STD-490A,  Specification  Practices,  4  June  1985. 

16.  MIL-STD-1521B,  Technical  Reviews  and  Audits  for  Systems,  Equipment,  and  Computer  Soft¬ 
ware,  4  June  1985. 

17.  AFOTEC  Instruction  99-101,  Management  of  Operational  Test  and  Evaluation,  1  October  1993. 


Figure  6.  Information  Sources  for  Software  Support  Life  Cycle  Process  Evaluation. 
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Project  Specific  Documents 

1.  Operational  Requirements  Document  (ORD). 

2.  Program  Management  Directive  (PMD). 

3.  Program  Management  Plan  (PMP). 

4.  Test  and  Evaluation  Master  Plan  (TEMP). 

5.  Request  for  Proposal  (RFP),  including  the  Statement  of  Work  (SOW)  and  the  Contract  Require¬ 
ment  Data  List  (CDRL). 

6.  Computer  Resources  Life  Cycle  Management  Plan  (CRLCMP), 

7.  Development  Test  and  Evaluation  Plans. 

8.  Operational  Test  and  Evaluation  Plans. 

9.  Contractor  Computer  Program  Development  Plan  (CPDP)  or  Software  Development  Plan  (SDP). 

10.  Contractor  Software  Configuration  Management  Plan  (SCMP). 

11.  Contractor  Software  Quality  Program  Plan  (SQPP). 

12.  Contractor  Software  Standards  and  Procedures  Manual  (SSPM). 

13.  Computer  Support  Integrated  Support  Document  (CRISP).  _ 

Figure  6,  Continued. 


6.2.  Note  that  the  correspondence  with  the 
letters  on  the  NCS  answer  sheet  (figure  5)  is 
as  follows  in  case  that  answer  sheet  is  used  to 
consolidate  the  question  responses: 

6  =  A 
5  =  B 

4  =  C 
3  =  D 

2  =  E 
1  =  F 

n/a=  H 

7.  Evaluation  Information  Sources.  The 

sources  of  information  to  determine  a  response 
to  a  question  can  be  categorized  in  many 
ways.  One  convenient  categorization  is: 

7.1.  Project  Documents.  Government,  con¬ 
tractor. 


7.2.  Regulations,  Directives,  Guidelines. 
Compliance,  internal. 

7.3.  Personnel.  Procurement,  development 
contractor,  operational  support,  interface 
working  groups. 

The  primary  information  sources  across  these 
categories  are  listed  in  figure  6.  The  termi¬ 
nology  for  some  of  the  questions  is  based  on 
the  DoD-STD-2167A,  Military  Standard  for 
Defense  System  Software  Development. 

8.  Summary  of  Evaluation  Philosophy. 

The  following  is  a  general  philosophy  which 
should  guide  you  in  answering  the  Software 
Support  Life  Cycle  Process  questions. 

8.1.  To  begin  with,  you  will  notice  that  the 
"questions"  are  really  statements  describing  a 
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particularly  desirable  attribute  of  software 
project  management  or  software  configuration 
management.  In  answering  the  question,  you 
will  have  to  quantify  your  qualitative  view¬ 
point  as  to  what  degree  that  desirable  attri¬ 
bute  is  reflected  in  the  system  under  evalu¬ 
ation.  The  average  of  the  answers  of  each  set 
of  questions  for  a  characteristic  then  provides 
the  basis  for  an  evaluation  of  the  contribution 
of  that  characteristic  to  the  software  support 
life  cycle  process  and  to  the  supportability  of 
the  system  software. 

8.2.  The  intent  of  this  evaluation  is  not  that  it 
is  completed  in  one  sitting  or  even  over  a 
week's  effort.  The  idea  is  that  this  evaluation 
is  a  running  history  of  the  processes  that 
affect  both  the  quality  of  the  software  products 
and  the  quality  of  the  support  resources 
needed  to  support  that  product.  Properly 
done,  this  evaluation  wiU  be  ongoing  through¬ 
out  our  involvement  of  a  system’s  develop 
ment  and  test.  There  is  plenty  of  room  on 
each  page  to  write  comments  on  your  findings 
and/or  thoughts  about  the  area  being 
addressed. 

8.3.  Your  answer  to  a  question  may  represent 
the  consensus  of  a  group  of  personnel  familiar 
with  the  characteristic  being  addressed  by  the 
question,  a  response  from  one  system  expert, 
or  your  response  derived  from  data  collected 
from  system  personnel  and  documentation. 

8.4.  You  should  establish  contacts  with 
knowledgeable  personnel  in  the  procurement 
activity,  development  contractor  activity,  and 
the  operational  support  activity  in  order  to 
identify  and  collect  information  which  would 
assist  you  in  answering  the  questions.  In 
most  Air  Force  programs,  the  procurement 
activity  will  be  from  one  of  the  product  centers 
of  Air  Force  Materiel  Command,  while  the 


operational  support  activity  will  be  one  of  the 
logistics  centers. 

8.5.  It  may  be  difficult  to  answer  all  ques¬ 
tions.  You  should  make  every  attempt  to 
obtain  enough  information  related  to  each 
question  to  make  a  reasonable  response. 
However,  you  should  not  respond  to  any  ques¬ 
tion  without  reasonable  rationale. 

8.6.  The  bottom  line  of  this  evaluation  is  "To 
what  extent  the  software  support  life  cycle 
process-in  particular,  software  project  man¬ 
agement  and  software  configuration-contrih- 
utes  to  the  supportability  of  the  system  soft¬ 
ware?"  Each  characteristic  evaluated  should 
he  considered  in  the  context  of  this  question. 

9.  How  to  Recommend  Changes.  This 
guide  is  not  a  perfect  test  tool;  we  will  change 
it  as  needed.  One  of  the  best  sources  for 
additional  information  to  be  included  or  areas 
that  should  be  revised  is  you,  the  one  who 
uses  this  evaluation  guide.  The  last  page  of 
this  pamphlet  contains  a  blank  Question  Data 
Sheet  you  can  use  to  recommend  changes. 
The  Question  Data  Sheet  may  be  used  to 
address  exact  questions  (fill  in  the  question 
number)  or  to  suggest  new  questions.  Send 
the  question  data  sheet  along  with  any 
additional  information  to: 

HQ  AFOTEC/SAS 

8500  Gibson  Blvd  SE 

Kirtland  AFB  NM  87117-5558 

Please  identify  yourself  and  the  circumstances 
which  led  to  your  recommendation,  and 
someone  from  the  Software  Analysis  Team 
will  contact  you  and  discuss  the  recommenda¬ 
tion  with  you. 


GEORGE  B.  HARRISON,  Major  General,  USAF 
Commander 


4  Attachments 

1.  Software  Support  Life  Cycle  Process  Question  Response  Guidelines 

2.  Summary  List  of  Questions 

3.  Questions  Number  List  by  Time  Phase 

4.  Glossary 

SOFTWARE  SUPPORT  LIFE  CYCLE  PROCESS 
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QUESTION  RESPONSE  GUIDELINES 

The  Software  Life  Cycle  Process  Questions  and  Response  Guidelines  are  presented  in  this 
attachment.  The  information  for  each  question  is  presented  on  one  page  and  consists  of: 

a.  Statement  of  the  evaluation  question. 

b.  Characteristic  identification. 

c.  Applicable  activity. 

d.  Explanation  of  the  question  as  appropriate. 

e.  Glossary  of  terms  as  appropriate. 

f.  Special  response  instructions  (if  any). 

g.  Response  rationale  to  be  completed  by  the  evaluator. 

h.  Response  score  to  be  completed  by  the  evaluator  (range  is  6  to  1). 

The  question  identification  information  is  at  the  top  right  of  each  page.  For  example,  "SCM(ID)  - 
001"  is  question  number  001  for  the  characteristic  Identification  (ID)  within  the  major  factor 
Software  Configuration  Management  (SCM).  In  addition,  each  set  of  characteristic  questions  (e.g., 
for  Identification)  is  preceded  by  a  one-page  description  of  the  characteristic  features. 

The  Software  Project  Management  and  Software  Configuration  Management  Guidelines  are 
presented  in  that  order. 
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ABBREVIATIONS  AND  ACRONYMS 


The  following  list  of  abbreviations  and  acronyms  is  frequently  used  in  the  questions. 


CCB 

configuration  control  board 

CDR 

critical  design  review 

CDRL 

contract  data  requirements  list 

CRISD 

computer  resources  integrated  support  document 

CRLCMP 

computer  resources  life  cycle  management  plan 

CRWG 

computer  resources  working  group 

CSC 

computer  software  component 

CSCI 

computer  software  configuration  item 

DID 

data  item  description 

DT&E 

development  test  and  evaluation 

ECP 

engineering  change  proposal 

FCA 

functional  configuration  audit 

FOT&E 

follow-on  operational  test  and  evaluation 

FQT 

formal  qualification  test 

HIPO 

hierarchy,  input,  process,  output 

HOL 

high  order  language 

HWCI 

hardware  configuration  item 

ICWG 

interface  control  working  group 

lOT&E 

initial  operational  test  and  evaluation 

ISA 

instruction  set  architecture 

IV&V 

independent  verification  and  validation 

ORD 

operations  requirements  document 

PCA 

physical  configuration  audit 

PDR 

preliminary  design  review 

PDSS 

postdeployment  software  support 

PID 

prime  item  development  (specification) 

PMD 

program  management  directive 

PMP 

program  management  plan 

PQT 

preliminary  qualification  test 

RFP 

request  for  proposal 

SCM 

software  configuration  management 

SCMP 

software  configuration  management  plan 

SCN 

specification  change  notice 

SDP 

software  development  plan 

SDR 

system  design  review 

SOW 

statement  of  work 

SPM 

software  project  management 

SQPP 

software  quality  program  plan 

SRR 

system  requirements  review 

SSPM 

software  standards  and  procedures  manual 

STP 

software  test  plan 

TEMP 

test  and  evaluation  master  plan 

TPWG 

test  planning  working  group 

TRR 

test  readiness  review 

WBS 

work  breakdown  structure 
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SOFTWARE  PROJECT  MANAGEMENT  PLANNING 


The  questions  SPM(PL)~001  through  SPM(PL)-032  address  adequacy  of  software  project 
management  planning  for  the  procurement,  development  contractor,  and  operational  support 
activities.  Project  management  planning  is  established  in  the  form  of  technical  documentation  that 
becomes  more  detailed  as  development  proceeds  and  more  refined  as  the  final  development  products 
are  evolved  during  postdeployment  support.  Three  levels  of  project  planning  are  generally  employed 
during  the  software  system's  life  cycle: 

(1)  Procurement  activity  project  planning. 

(2)  Development  contractor  activity  project  planning. 

(3)  Operational  support  activity  project  planning. 

The  procurement  activity  project  plans  include: 

(1)  Program  Management  Plan  (PMP). 

(2)  Test  and  Evaluation  Master  Plan  (TEMP). 

(3)  RFP/SOW/CDRL  package. 

(4)  DT&E  Plans. 

(5)  OT&E  Plans. 

(6)  Computer  Resources  Life  Cycle  Management  Plan  (CRLCMP). 

The  development  contractor  activity  project  plans  include: 

(1)  Software  Development  Plan  (SDP). 

(2)  Software  Configuration  Management  Plan  (SCMP). 

(3)  Software  Quality  Program  Plan  (SQPP). 

(4)  Software  Standards  and  Procedures  Manual  (SSPM). 

(5)  Software  test  planning.  (Plan,  Procedures,  Description,  Acceptance). 

(6)  Computer  Resources  Integrated  Support  Document  (CRISD). 

The  operational  support  activity  project  plans  include: 

(1)  CRLCMP. 

(2)  Software  Configuration  Management  Plan  (SCMP). 

(3)  Software  Support  Management  Plan  (SSMP). 

(4)  Other  agreements  (e.g..  Memorandums  of  Agreement). 

The  adequacy  of  software  project  management  planning  with  respect  to  the  area  of  software 
supportability  is  mostly  a  matter  of  early  identification  of  software  supportability  characteristics  in 
planning  documents,  procurement  requiring  software  supportability  characteristics,  development 
contractor  implementing  the  characteristics,  and  operational  support  transitioning  early  life  cycle 
concepts  and  continuing  the  evolution  process  through  the  postdeployment  life  cycle  phase. 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  --  001 

QUESTION:  Planning  for  computer  resources  has  been  adequate  with  respect  to  acquisition, 
development,  logistics,  and  training. 

ACTIVITY(IES):  Procurement  and  Operational  Support 

EXPLANATION:  The  primary  procurement  planning  documents  include  the  PMP,  TEMP, 
System/Segment  Specification,  and  CRLCMP,  The  computer  resources  includes  the  hardware, 
software,  personnel,  procedures,  facilities,  schedule,  budget,  and  so  forth.  All  aspects  of  acquisition, 
development,  logistics  support,  and  training  must  be  planned.  The  Milestones  I,  II,  and  III  provide 
major  event  check  points  for  analysis  and  review  of  plans.  Analysis  should  always  be  conducted  with 
respect  to  operational  requirements  and  the  results  integrated  back  into  the  "living"  plans.  Plans  for 
measuring  software  quality  attributes,  in  particular  software  support  ability,  are  required. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  002 

QUESTION:  Procurement  planning  for  computer  resources  has  been  consistent  with  the  system 
development  and  acquisition  plan. 

ACTIVITY(IES):  Procurement  and  Operational  Support 

EXPLANATION:  The  primary  system  development  and  acquisition  plan  is  the  PMP.  The  CRLCMP 
should  be  derived  from  the  allocation  of  the  system  requirements  in  the  PMP  to  computer  resources. 
The  CRLCMP  identifies  computer  resource  acquisition  and  life  cycle  support  requirements.  The 
CRLCMP  reflects  the  software  development  and  support  approach  for  the  system  and  is  evolved  as  a 
living  document  throughout  the  system  life  cycle. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  003 

QUESTION:  Planning  for  computer  resources  has  been  based  on  an  acquisition  schedule  with 
adequately  specified  milestones. 

ACTIVITY(IES):  Procurement  and  Operational  Support 

EXPLANATION:  The  plans  should  be  based  on  a  realistic  acquisition  schedule.  Major  Milestones  I, 
II,  and  III  should  be  specified  as  well  as  the  major  review  and  audit  points  such  as  SDR,  SRR,  PDR, 
CDR,  TRR,  FCA,  and  PCA.  The  transition  and  turnover  dates  should  also  realistically  reflect  the 
risks  in  acquiring  and  supporting  such  a  system.  Studies  and  analysis  should  have  been  performed 
prior  to  the  Engineering  and  Manufacture  Development  effort  in  order  to  determine  what  a  realistic 
schedule  should  be. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  004 

QUESTION:  Computer  resources  have  been  adequately  addressed  as  major  considerations  at 
procurement  reviews,  audits,  and  management  evaluations. 

ACTIVITY(IES):  All 

EXPLANATION:  Typical  procurement  reviews,  audits,  and  management  evaluation  involve 
participation  of  all  activities.  The  degree  of  participation  depends  on  the  particular  event. 
Feasibility  studies,  tradeoff  analysis,  protot5q)e  developments,  and  milestone  decision  points 
determine  how  thoroughly  computer  resources  have  been  addressed. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4,3,2, 1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  005 

QUESTION:  Planned  computer  resources  have  been  analyzed  adequately  by  the  program  office  to 
ensure  conformance  with  stated  operational  and  support  requirements. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  Methods  of  analysis  include  feasibility  studies,  tradeoff  studies,  risk  analysis,  and 
prototype  development.  These  methods  usually  occur  during  Concept  Exploration  and/or 
Demonstration  and  Validation  phases  prior  to  the  Engineering  and  Manufacture  phase.  During 
Engineering  and  Manufacture  Development,  an  IV&V  fvmction  can  assist  in  such  analysis. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4,3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  006 

QUESTION:  Procurement  planning  software  quality  attributes  has  been  adequately  emphasized 
throughout  the  software  life  cycle  acquisition. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  Software  quality  attributes  should  be  a  major  consideration  in  the  initial 
planning  of  software  requirements.  This  emphasis  should  be  continued  throughout  the  system  and 
software  life  cycle  phases.  Software  quality  requirements  and  responsibilities  should  be  defined  in 
the  CRLCMP.  Procedures  should  be  developed  and  implemented  to  ensure  proper  assessment  of 
computer  resources  throughout  the  system  life  cycle.  The  procurement  activity  should  develop 
assessment  procedures  to  ensure  that  the  computer  software  will  meet  management  policies  and 
appropriate  regulations  throughout  the  system  life  cycle.  Computer  software  should  be  assessed 
continuously  by  means  of  reviews,  audits,  verification  vahdation  testing,  and  other  enforcement 
activities. 

GLOSSARY: 

Software  Quality  Attributes.  Reliability,  Supportability,  Maturity,  Efficiency,  and  so  forth. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  007 

QUESTION:  Margins  for  reserve  computer  resource  capacity  to  provide  for  later  product 

improvements  are  adequate. 

ACTIVITY(IES):  All 

EXPLANATION:  Requirements  for  margins  and  the  initial  values  are  established  by  the 
procurement  and  operational  support  activity  in  the  PMD,  RFP/SOW,  and  System/Segment 
Specification.  These  margins  are  then  evolved  throughout  the  Engineering  and  Manufacture  by  the 
development  contractor  activity.  Margins  should  be  established  for  memory,  external  storage,  task 
utilization,  terminal  usage,  performance  parameters,  and  so  forth.  Typical  guidelines  are  to  leave  30 
to  50  percent  of  the  total  resource  capacity  as  reserve  dependent  upon  the  resource  and  the 
particular  application.  The  margin  of  reserve  is  very  important  for  software  supportability  since 
changes  will  usually  require  consumption  of  some  of  the  reserve. 

GLOSSARY: 

Margin.  The  difference  between  the  total  available  capacity  of  a  resource  and  the  actual  amount 
used  divided  by  100. 

RESPONSE  INSTRUCTIONS: 

F/1:  50%  or  less  of  required  reserve  capacity  is  available  for  at  least  one  of  the  resources,  or  no 
margin  requirements  have  been  established 

E/2:  50%  to  59%  of  required  reserve  capacity  is  available  for  at  least  one  of  the  resources 
D/3:  60%  to  69% 

C/4:  70%  to  79% 

B/5:  80%  to  89% 

A/6:  90%  to  100%  of  required  reserve  capacity  is  available  for  all  resources 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  008 

QUESTION;  Acceptable  techniques  have  been  used  to  estimate  and  monitor  software  costs 
throughout  the  system  life  cycle. 

ACTIVITY(IES):  All 

EXPLANATION:  It  is  necessary  that  a  standard  technique  (e.g.,  COCOMO  model)  be  used  to 
estimate  software  costs  throughout  the  system  life  cycle.  It  is  probably  more  important  that  some 
technique  be  consistently  used  and  the  results  carefully  monitored  than  which  technique  is  used. 
Each  activity  should  have  some  method  that  is  used  and  a  way  to  correlate  cost  results  from  the 
other  activities  with  their  results.  A  cost/schedule  risk  analysis  should  be  done  at  each  of  the  major 
project  life  cycle  decision  points.  The  software  costs  are  usually  related  to  the  related  WBS  tasks  in 
order  to  accomplish  the  cost/schedule  risk  analysis. 

GLOSSARY: 

COCOMO.  Constructive  Cost  Model  (B.  Boehm  of  TRW) 

Software  Cost.  The  resources  consumed  to  develop  and  support  software  throughout  its  life 
cycle. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  009 

QUESTION:  The  CRLCMP  contains  adequate  specifications  of  the  acquisition  requirements  for 
computer  resources. 

ACTIVITY(IES):  Procurement  and  Operational  Support 

EXPLANATION:  During  the  development  phase,  the  CRLCMP  serves  to  define  the  development 
plan  and  to  identify  the  computer  resources  necessary  to  support  the  system  after  deployment.  After 
managing  the  support  of  the  system's  computer  resources,  the  procurement  activity  should  begin 
preparing  the  CRLCMP  with  the  help  of  the  operational  support  activity,  with  completion  no  later 
than  Milestone  II  or  equivalent.  At  this  point,  the  CRLCMP  should  focus  on  plans  for  developing  the 
computer  resources,  including  computer  resources  needed  for  system  support.  As  the  system 
progresses  into  the  Engineering  and  Manufacture  phase,  the  CRLCMP  should  be  expanded  to 
provide  a  comprehensive  plan  for  support  of  computer  resources.  By  Milestone  III  or  equivalent,  the 
CRLCMP  should  contain  a  plan  for  transitioning  computer  resource  responsibilities  from  the 
procurement  activity  to  the  operational  support  activity. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  010 

QUESTION:  The  CRLCMP  adequately  addresses  the  responsibilities  and  procedures  to  ensure 
proper  software  configuration  management  throughout  the  system  life  cycle, 

ACTIVITY(IES):  Procurement  and  Operational  Support 

EXPLANATION:  Responsibilities  should  be  defined  in  the  CRLCMP,  and  procedures  should  be 
developed  and  implemented  to  ensure  proper  control  of  computer  resources  throughout  the  system 
life  cycle.  Computer  hardware  and  software  should  be  identified,  specified,  and  managed  as 
configuration  item.  The  mechanism  for  controlling  computer  hardware  and  software  changes  is  the 
documentation  for  each  configuration  item,  and  it  is  the  responsibility  of  the  system  configuration 
manager  within  the  procurement  or  operational  support  activity  to  ensure  that  this  documentation  is 
accurate  and  current.  The  Configuration  Control  Board  (CCB)  should  be  the  primary  authority  for 
approving  hardware  and  software  changes  to  the  existing  baseline.  The  CCB  membership  should  be 
determined  by  the  procurement  and/or  the  operational  support  activity. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 


AFOTECPAM  99-102  Volume  2  Attachment  1  1  August  1994 


31 


QUESTION  DATA  SHEET 

Question  Number  SPM(PL)  -  Oil 

QUESTION:  The  project  management  responsibility  for  integrating  computer  resources  into  a 
system  has  remained  centralized  throughout  the  life  of  the  system. 

ACTIVITY(IES):  All 

EXPLANATION:  The  minimum  requirement  is  that  each  activity's  project  management 

responsibility  remain  with  the  same  organizational  element.  For  example,  the  implementing 
command  (product/logistics  centers),  development  contractor,  and  using  command  remain  the  same. 
A  more  important  aspect  of  this  centralization  is  that  the  lower  level  organizational  structure 
(including  personnel)  within  each  activity  should  remain  intact  without  fragmentation  or  major 
variance  of  responsibility  over  all  life  cycle  phases. 

GLOSSARY: 

Centralized.  Located  within  the  same  organizational  element. 

RESPONSE  INSTRUCTIONS: 

F/1;  Under  50%  of  the  project  management  across  all  activities  and  all  phases  has  remained 
centralized 

E/2;  50%  to  59% 

D/3:  60%  to  69% 

C/4:  70%  to  79% 

B/5:  80%  to  89% 

A/6;  90%  to  100% 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3,2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(PL)  -  012 

QUESTION:  The  CRWG  organization  has  been  adequate  throughout  the  system  life  cycle. 
ACTrVlTY(IES):  Procurement  and  Operational  Support 

EXPLANATION:  If  not  already  established  during  the  Concept  Exploration  phase,  the  CRWG 
should  be  formed  early  in  the  Demonstration  and  Validation  phase  to  assist  the  program  manager  in 
planning  and  implementing  software  issues,  activities,  and  functions.  During  acquisition,  the 
implementing  command  chairs  and  manages  the  CRWG.  The  CRWG  should  also  include  the  using 
command,  supporting  logistics  centers,  and  perhaps  representatives  of  other  organizations 
responsible  for  DT&E,  OT&E,  and  training. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  013 

QUESTION:  The  CRWG  has  had  clearly  specified  responsibilities  and  appropriate  authority  to 
implement  those  responsibilities  throughout  the  system  life  cycle. 

ACTIVITY(IES):  Procurement  and  Operational  Support 

EXPLANATION:  The  CRWG  supports  the  program  manager  in  developing  the  CRLCMP.  Also,  the 
CRWG  recommends  alternatives  in  areas  such  as  documentation  requirements,  software  security 
requirements,  IV&V,  standard  equipment,  standard  HOLs,  standard  ISAs,  and  margins  for  reserve 
computer  capacity.  The  CRWG  identifies  the  computer  resources  and  facilities  required  to  support 
the  system  throughout  the  system  life  cycle.  For  programs  with  interservicing  potential,  the  CRWG 
includes  members  from  each  organization  affected  by  interservicing.  The  CRWG  analyzes 
interservicing  potential  to  support  the  program  manager's  decision  concerning  a  joint  service  facility. 
This  analysis  should  consider  operational  needs,  life  cycle  costs,  technical  capability,  and  service- 
unique  standards  for  computer  resources.  Without  the  proper  authority  to  implement  its  decisions 
and  have  its  recommendations  acted  upon,  the  CRWG  will  be  deficient. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE' 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(PL)  -  014 

QUESTION:  The  CRWG  has  properly  ensured  that  computer  resources  comply  with  established 
policy,  procedures,  plans,  and  standards. 

ACTIVITY(BES);  Procurement  and  Operational  Support 

EXPLANATION:  The  CRWG  assists  in  ensuring  that  computer  resources  comply  with  established 
policy,  procedures,  plans,  and  standards.  The  CRWG  continuously  supports  the  procurement  activity 
in  planning  the  computer  resource  life  cycle  and  evaluating  developed  computer  resources.  The 
CRWG  recommends  updates  to  the  CRLCMP  to  ensure  that  acquisition,  user,  and  support 
requirements  are  satisfied.  The  CRWG  evaluates  computer  software  plans,  products,  and  proposed 
changes  to  ensure  compatibility  with  the  CRLCMP  and  consistency  with  policies  and  procedures. 
The  CRWG  also  supports  the  procurement  activity  in  the  resolution  of  issues  such  as  documentation 
requirements  and  support  agreements.  If  computer  resources  development  is  part  of  an  interservice 
program,  then  the  interservice  CRWG  verifies  that  the  required  computer  resources  and  operational 
support  capabilities  are  available  to  support  the  system.  Before  Milestone  III  or  equivalent,  the 
CRWG  should  develop  interservice  procedures  for  operational  support  of  the  system.  In  addition,  the 
CRWG  ensures  that  the  joint  service  operational  support  activity  participates  in  the  Engineering  and 
Manufacture  Development  phase,  thereby  acquiring  the  necessary  familiarity  and  experience  to 
support  the  system  on  completion  of  development. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  015 

QUESTION:  Software  quality  assessment  procedures  have  been  adequately  defined  to  meet 
management  policies  and  appropriate  regulations,  conform  to  standards,  and  meet  performance  and 
quality  requirements  throughout  the  system  life  cycle. 

ACTIVITYdES):  All 

EXPLANATION:  Software  quality  requirements  and  responsibilities  shall  he  defined  in  the 
CRLCMP.  Procedures  should  be  developed  and  implemented  to  ensure  proper  assessment  of 
computer  resources  throughout  the  system  life  cycle.  The  procurement  activity  develops  assessment 
procedures  to  ensure  that  the  computer  software  will  meet  management  policies  and  appropriate 
regulations,  conform  to  standards,  and  meet  performance  and  quality  requirements  throughout  the 
system  life  cycle.  Computer  software  should  be  assessed  continuously  by  means  of  reviews,  audits, 
verification  validation  testing,  and  other  enforcement  activities.  The  primary  software  quality 
standard  is  DoD-STD-2168. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6,5,4, 3,2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(PL)  >016 

QUESTION:  Planning  for  DT&E  of  computer  resources  has  been  adequate  throughout  the  system 
life  cycle. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  primary  high-level  planning  document  for  DT&E  is  the  TEMP,  DT&E 
descriptions  in  the  TEMP  should  concentrate  on  technical  goals,  thresholds,  and  objectives.  At  each 
review  phase,  the  essential  questions  should  continue  to  be  whether  objectives  were  met,  degree  of 
confidence  in  results,  and  specific  system  behavior  leading  to  observed  anomalies.  The  detailed 
DT&E  plans  supplement  the  TEMP  and  provide  insight  into  the  specific  software  and  system 
integration  tests  and  procedures  that  are  planned. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 


AFOTECPAM  99-102  Volume  2  Attachment  1  1  August  1994 


37 


QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -■  017 

QUESTION;  Planning  for  OT&E  of  computer  resources  has  been  adequate  throughout  the  system 
life  cycle. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  primary  high-level  planning  document  for  OT&E  is  the  TEMP.  More 
detailed  OT&E  plans  supplement  the  TEMP.  The  types  of  OT&E  are  lOT&E  and  FOT&E.  lOT&E 
is  the  first  test  of  a  complete  system  and  support  elements  in  an  operational  environment.  lOT&E 
provides  an  early  over-the-shoulder  effort  during  the  Demonstration  and  Validation  phase  of  the 
system  life  cycle.  The  purpose  of  the  early  lOT&E  is  to  provide  an  operational  input  to  the  initial 
development  program,  ensure  the  coupling  of  requirements  to  the  development  program,  develop  an 
interface  between  developer  and  user,  and  refine  for  Engineering  and  Manufacture  Development;  a 
later  lOT&E  is  conducted  prior  to  the  production  decision.  FOT&E  is  conducted  when  the  system  is 
fully  deployed  in  order  to  assess  full  system  capability.  Specific  objectives,  measures  of  effectiveness, 
and  measures  of  performance  should  be  addressed  in  the  OT&E  planning  documents. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(PL)  -  018 

QUESTION;  Software  standards  have  been  adequately  specified  throughout  the  software  life  cycle. 

ACTIVITY(IES):  All 

EXPLANATION:  The  RFP/CDRL/SOW  reflects  required  software  standards  as  specified  by  the 
procurement  activity.  The  standards  should  apply  to  all  aspects  of  the  software  development  and 
support.  Typical  standards  are  DoD-STD-2167A,  DoD-STD-2168,  MIL-STD-483A,  MIL-STD-1521B, 
DoD-STD-1815A,  and  various  ANSI/IEEE  software  engineering  standards.  The  development 
contractor  must  comply  with  the  procurement  requirements  through  internal  standards  and 
conventions.  The  operational  support  activity  contributes  to  the  recommendations  on  the  use  of 
required  standards  and  sets  internal  standards  and  conventions  for  use  during  the  postdeployment 
support  of  the  software. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  019 

QUESTION:  The  planning  for  organic  and/or  contractor  support  during  postdeployment  software 
support  has  been  adequate. 

ACTIVITY(IES):  Operational  Support 

EXPLANATION:  Software  supportability  characteristics  include  software  product  maintainahility, 
software  support  resources,  and  software  life  cycle  processes.  Plans  for  organic  and/or  contractor 
support  and  the  evaluation  of  whether  such  support  will  be  adequate  relative  to  software 
supportability  characteristics  should  he  contained  in  the  CRLCMP . 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(PL)  -  020 

QUESTION:  Contractual  documents  have  explicitly  established  government  rights  to  all  computer 
resources  required  to  develop,  operate,  simulate,  test,  and  support  the  software. 

ACTIVITY(IES):  Procurement  and  Operational  Support 

EXPLANATION:  Contractual  documents  should  clearly  establish  government  rights  to  all 
computer  resources  required  to  support  the  softwEtre.  The  rights  may  have  proprietary  clauses  that 
must  be  carefully  understood  by  all  parties.  The  appHcation  software,  system  software,  and  test 
software  should  be  considered. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6,5,4,3,2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  021 

QUESTION:  Planning  for  risk  analysis  to  identify  areas  of  computer  resource  risk  has  been 
adequate. 

ACTIVITY(IES):  Procurement  and  Operational  Support 

EXPLANATION:  A  common  source  of  operational  software  problems  is  the  difficulty  of  maintaining 
and  supporting  the  software  once  it  is  deployed.  The  technology  used  to  design  and  implement  the 
software  may  significantly  affect  its  ability.  Danger  signals  may  include  the  use  of  proprietary  tools 
and  techniques  that  will  not  be  available  to  engineers  after  system  delivery.  Alternatively,  there 
may  be  unique  aspects  of  the  design  effort  that  positively  affect  subsequent  life  cycle  cost  and  effort. 
One  approach  to  reducing  long-term  life  cycle  risks  is  to  enforce  the  use  of  common  technology 
throughout  the  development  and  operation  of  the  software.  It  is  not  uncommon  for  the  project  office 
to  supply  tools  and  support  software  to  the  development  contractor  to  ensure  commonality. 
However,  care  should  he  exercised  to  avoid  government  liability  in  cases  of  inadequate  government- 
furnished  tools.  Ideally,  life  cycle  characteristics  of  operational  significance  should  be  listed  as 
required  characteristics  of  the  system,  and  tests  should  be  planned  to  address  the  issues  that  arise 
from  these  characteristics. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6,5,4,3,2,1  =  COMPLETELY  DISAGREE) 


42 


AFOTECPAM  99-102  Volume  2  Attachment  1  1  August  1994 


QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  022 

QUESTION:  A  mission/function  matrix  (or  equivalent)  clearly  identifies  primary  functional 
capabilities  to  be  implemented  by  the  software. 

ACTIVITY(IES):  Procurement  and  Operational  Support 

EXPLANATION:  A  mission/function  matrix  (or  equivalent  narrative)  is  the  primary  source  of 
information  about  how  the  system  capabilities  have  been  partitioned  between  hardware  and 
software.  These  partitions  will  be  important  in  determining  required  characteristics,  in  defining 
error/failure  criteria,  and  in  isolating  and  correcting  deficiencies  noted  during  testing.  Therefore,  it 
may  be  important  to  determine  that  proper  engineering  studies  have  led  to  the  establishment  of 
these  partitions.  An  understanding  of  the  sources  of  risk  in  each  of  the  software-implemented 
functions  identified  in  the  mission/function  matrix  is  an  essential  part  of  the  overall  risk  assessment. 

GLOSSARY: 

Mission/Function  Matrix.  Correspondence  relating  primary  functional  capabilities  that  must  be 
demonstrated  by  testing  to  the  mission  to  be  performed  and  the  concept  of  operation. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  023 

QUESTION:  Plemning  for  interoperability  with  other  systems  has  been  adequately  addressed. 
ACTIVITYdES):  Procurement  and  Operational  Support 

EXPLANATION:  When  required,  systems  must  be  interoperable  with  other  systems  employed  by 
the  US  and  Allied  military  forces.  Interoperability  requirements  should  be  identified,  defined, 
validated,  and  included  in  appropriate  planning  documentation  prior  to  the  end  of  the 
Demonstration  and  Validation  phase.  Both  development  and  postdeployment  support  concerns 
should  be  addressed  and  organizational  responsibilities  defined. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

A/6:  No  interoperability  requirements  exist  for  the  subject  system 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6,5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(PL)  -  024 

QUESTION:  Prior  to  each  system  milestone,  interservicing  potential  and  life  cycle  cost  implications 
of  software  support  options  have  been  appropriately  addressed. 

ACTIVITY(IES):  Procurement  and  Operational  Support 

EXPLANATION:  Before  each  system  milestone,  interservicing  potential  should  be  reviewed  and 
the  management  and  life  cycle  cost  implications  of  major  software  support  options  should  be 
analyzed.  This  analysis  should  also  consider  impact  on  operational  needs,  configuration 
management,  and  system  integration.  For  interservice  systems,  the  CRWG  should  be  an  interservice 
CRWG  that  includes  representatives  from  all  cognizant  organizational  elements.  The  CRWG 
(interservice  or  single  service)  should  ensure  that  analysis  is  performed  to  determine  the  optimum 
support  approach,  document  this  analysis,  and  make  recommendations  to  the  procurement  activity 
concerning  the  support  approach. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS; 

RESPONSE  RATIONALE; 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  025 

QUESTION;  The  procurement  and  operational  support  planning  dociunents  have  been  adequately 
updated  as  living  documents  throughout  the  system  life  cycle. 

ACTIVITY(IES):  Procurement  and  Operational  Support 

EXPLANATION:  Some  of  the  documents  that  should  be  continually  updated  include  the  TEMP, 
CRLCMP,  and  the  DT&E  and  OT&E  plans.  It  is  important  that  these  documents  are  working  plans 
and  as  such  track  closely  the  progress  and  current  status  of  the  system. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(PL)  -  026 

QUESTION:  The  principles  and  methodologies  provided  in  the  regulations  have  been  appropriately 
incorporated  into  the  software  test  and  evaluation  plans. 

ACTIVITY(IES):  All 

EXPLANATION:  Typical  procurement  and  operational  support  regulations  include  DoD  5000.2, 
APR  800-14,  AFR  80-14,  and  APR  55-43.  Typical  development  contractor  compliance  regulations 
include  DoD-STD-2167A,  DoD-STD-2168,  and  MIL-STD-1521B.  The  software  test  and  evaluation 
plans  are  primarily  contained  in  the  TEMP,  DT&E  plans,  OT&E  plans,  IV&V  plans,  and 
development  contractor  plans. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6,5,4,3,2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  027 

QUESTION:  Planning  for  systematic,  quantitative,  and  objectively  reportable  software  tests  has 
been  adequate. 

ACTIVITY(IES):  All 

EXPLANATION:  It  should  be  apparent  from  the  description  of  the  software  tests  conducted  and 
their  results  whether  or  not  previous  goals  have  been  met  and  test  objectives  have  been  satisfied. 
Vague  references  to  "successful  software  results"  or  "no  problems  with  the  software"  should  not  be 
acceptable.  In  order  to  evaluate  the  progress  of  software  testing  to  date,  there  must  be  explicit 
reference  to  (1)  a  systematic,  scientifically  sound  approach  to  carrying  out  the  test,  (2)  the 
relationship  between  the  systematic  test  approach  and  the  test  objectives  for  the  current  phase,  (3) 
the  results  of  the  test,  and  (4)  the  plans  for  resolution  of  errors. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  028 

QUESTION:  Planning  for  sharing  of  software  test  results  across  life  cycle  phases  and  among  test 
organizations  has  been  adequate. 

ACTIVITYdES):  All 

EXPLANATION:  Successful  test  and  evaluation  requires  involvement  and  data  sharing  across  all 
life  cycle  phases  for  test  organizations:  DT&E,  OT&E,  IV&V,  and  the  development  contractor.  The 
TEMP  is  the  primary  high-level  planning  document  for  software  test  and  evaluation  with 
appropriate  references  to  all  the  organizational  elements'  plans.  The  development  contractor  activity 
is  responsible  for  developing  test  plans  and  procedures  to  effect  an  appropriate  sharing  (as  defined 
contractually  in  the  SOW)  of  test  results. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  029 

QUESTION:  Tracking  of  computer  resource  utilization  has  been  adequately  planned. 
ACTIVITY(IES):  All 

EXPLANATION:  The  procurement  activity  must  plan  to  have  tracking  data  collected.  This  plan  is 
put  into  action  through  appropriate  contract  requirements.  The  development  contractor  must  plan 
to  implement  data  collection  procedures  that  satisfy  the  contractual  requirements  as  a  minimum. 
Normally,  more  detailed  data  must  be  collected  by  the  development  contractor  in  order  to  properly 
derive  the  required  contract  data  and  adequately  manage  the  development  effort.  Typical  data  to  be 
collected  include  memory,  central  processing  unit  usage,  and  input/output  throughput.  The 
maximum  and  minimum  values  and  actual  resource  utilization  data  values  are  required.  These  data 
are  necessary  to  determine  if  required  margins  of  reserve  will  be  met.  See  AFP  800-48  for  more 
information. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  030 

QUESTION:  The  project  software  budget/cost  variance  (budgeted  -  actual)  appears  to  be  reason¬ 
able. 

ACTIVITY(IES):  All 

EXPLANATION:  Depending  on  the  perspective,  all  activities  are  required  to  adequately  manage 
the  software  budget/cost.  At  each  major  management  review  for  the  activity,  the  variance  between 
what  was  budgeted  and  what  has  actually  been  consumed  should  be  analyzed.  Based  on  the  known 
contractual  changes  in  requirements,  an  assessment  should  be  made  whether  the  variance  in  cost  is 
within  certain  limits.  The  limits  should  probably  be  established  during  the  Demonstration  and 
Validation  phase  through  a  risk  analysis.  See  AFP  800-48  for  more  information. 

GLOSSARY: 

Budget/Cost  Variance.  The  budgeted  cost  of  software  task  work  (WBS)  performed  minus  the 
actual  cost  of  software  task  work  performed. 

Percentage  Variance.  The  Budget/Cost  Variance  divided  by  the  actual  cost  of  software  task 
work  performed  times  100. 

RESPONSE  INSTRUCTIONS: 

F/1:  plus  or  minus  25%  or  more  Percentage  Variance 

E/2:  plus  or  minus  20%  to  25%  Percentage  Variance 
D/3:  plus  or  minus  15%  or  20%  Percentage  Variance 
C/4:  plus  or  minus  10%  or  15%  Percentage  Variance 
B/5:  plus  or  minus  5%  or  10%  Percentage  Variance 
A/6:  plus  or  minus  0%  or  5%  Percentage  Variance 

Underruns  can  be  as  big  an  indicator  of  a  problem  as  an  overrun,  but  each  case  should  be  carefully 
considered  on  its  own  merits. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  031 

QUESTION:  The  project  software  schedule/cost  variance  (consumed  -  scheduled)  appears  to  be 
reasonable. 

ACTIVITY(IES):  All 

EXPLANATION:  Depending  on  the  perspective,  all  activities  are  required  to  adequately  manage 
the  software  schedule/cost.  At  each  major  management  review  for  the  activity,  the  variance  between 
what  was  consumed  and  what  has  actually  been  scheduled  should  be  analyzed.  Based  on  the  known 
contractual  changes  in  requirements,  an  assessment  should  be  made  whether  the  variance  in  cost  is 
within  certain  limits.  The  limits  should  probably  be  established  during  the  Demonstration  and 
Validation  phase  through  a  risk  analysis.  See  AFP  800-48  for  more  information. 

GLOSSARY: 

Schedule/Cost  Variance,  The  budgeted  cost  of  software  task  work  (WBS)  performed  minus  the 
budget  cost  of  software  task  work  scheduled. 

Percentage  Variance.  The  Schedule/Cost  Variance  divided  by  the  actual  cost  of  software  task 
work  performed  times  100. 

RESPONSE  INSTRUCTIONS: 

F/1:  plus  or  minus  25%  or  more  Percentage  Variance 

E/2:  plus  or  minus  20%  to  25%  Percentage  Variance 

D/3:  plus  or  minus  15%  or  20%  Percentage  Variance 
C/4:  plus  or  minus  10%  or  15%  Percentage  Variance 

B/5:  plus  or  minus  5%  or  10%  Percentage  Variance 

A/6:  plus  or  minus  0%  or  5%  Percentage  Variance 

Schedule  decreases  can  be  as  big  an  indicator  of  a  problem  as  an  increases,  but  each  case  should  be 
carefully  considered  on  its  own  merits. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PL)  -  032 

QUESTION:  The  cost  and  schedule  contractual  reporting  requirements  appear  to  be  adequate. 

ACTIVITYdES):  All 

EXPLANATION:  The  data  necessary  to  determine  the  cost  variance  and  schedule  variEmce 
information  in  questions  SPM(PL)-30  and  SPM(PL)-031  are  a  minimum  requirement.  DoDI  7000.2 
is  the  policy  for  financial  management  reporting  for  the  development  contractor  activity  (appropriate 
size  and  types  of  contracts).  AFSCP  173-5  implements  the  pohcy  of  DoDI  7000.2  and  provides 
specific  criteria  for  the  development  contractor  activity. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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SOFTWAKE  PROJECT  MANAGEMENT  ORGANIZATION  STRUCTURE 

The  questions  SPM(OS)-001  through  SPM(OS)-O19  address  adequacy  of  software  project 
management  organization  structure  for  the  procurement,  development  contractor,  and  operational 
support  activities.  Project  management  organization  structure  is  established  so  that  the  functional 
requirements  of  a  project  can  be  more  effectively  accomplished.  This  organization  structure  includes 
the  physical  relationship  among  the  organization  elements,  the  logical  structure  that  relates  the 
organization  elements  to  the  project's  functional  requirements,  and  the  stability  and  capability  of  the 
personnel  who  staff  the  organization. 

Characteristics  that  affect  the  organization  structure  are  number  of  internal  interfaces,  size  of 
organization,  stability  of  the  physical  structure,  continuity  and  capability  of  project  personnel,  and 
capability  of  the  physical  organization  to  effectively  handle  responsibilities  inherent  in  the  software 
project  work  breakdown  structure  tasks. 
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QUESTION  DATA  SHEET 


Question  Number  SPM(OS)  -  001 

QUESTION;  The  software  requirements  have  been  adequately  allocated  to  elements  of  a  Work 
Breakdown  Structure  (WBS). 

ACTIVITY(IES):  All 

EXPLANATION:  The  WBS  should  clearly  indicate  those  task  areas  where  software  related 
requirements  are  addressed.  A  traceability  matrix  should  clearly  indicate  how  those  requirements 
have  been  mapped  to  the  WBS  elements.  All  activities  (procurement,  development  contractor, 
operational  support)  are  involved  in  some  aspect  of  the  WBS,  software  requirements,  and  the 
allocation  process. 

GLOSSARY: 

Software  Requirements.  Contractual  system  requirements  that  have  been  allocated  to  software 
functions.  These  are  usually  found  in  a  system/segment  specification  and/or  a  functional 
development  specification. 

Work  Breakdown  Structure.  Product-oriented  hierarchical  definition  of  all  tasks  to  be 
performed  and  accounted  for  in  the  course  of  the  project  development. 

RESPONSE  INSTRUCTIONS: 

F/1:  No  WBS  exists,  no  software  requirements  specification  exists,  and/or  there  is  no  evidence  of 

an  allocation  of  software  requirements  to  WBS  elements. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(OS)  -  002 

QUESTION:  The  software  related  tasks  are  clearly  identified  in  the  WBS, 

ACTIVITY(IES):  All 

EXPLANATION:  Typical  software  tasks  include  project  management,  development,  documenta¬ 
tion,  test,  environment  evolution,  configuration  management,  quality  assurance,  acceptance,  and 
transfer.  All  software-related  tasks  should  be  separately  identified  from  hardware-related  tasks  as 
much  as  is  possible.  The  more  accurately  such  tasks  can  be  tracked  and  reported,  the  more  likely 
early  problem  areas  can  be  identified  and  resolved. 

GLOSSARY: 

Software  Task.  Project  task  whose  primary  function  is  related  to  the  production  and/or  delivery 
of  a  software  product. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(OS)  -  003 

QUESTION:  The  key  project  personnel  and  their  assignments  in  relation  to  the  WBS  software- 
related  tasks  are  clearly  identified. 

ACTIVITY(IES):  All 

EXPLANATION:  In  order  to  properly  identify  responsibilities  and  communication  channels,  it  is 
necessary  to  have  the  key  project  personnel  for  all  activities  identified  along  with  their  areas  of 
responsibility  and  their  relationship  to  the  WBS. 

GLOSSARY: 

Key  Project  Personnel.  Project  managers,  task  managers,  task  technical  leaders  for  all 
activities. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(OS)  -  004 

QUESTION:  The  coordination  of  modifications  to  the  WBS  among  all  activities  has  been  adequate. 
ACTIVITY(IES):  All 

EXPLANATION:  Whenever  the  WBS  is  modified,  there  is  potential  to  have  a  significant  effect  on 
all  activities.  This  effect  may  be  in  the  form  of  a  modification  in  schedule,  level  of  effort,  deliverable 
product,  functional  capability,  and/or  system  interface.  A  mechanism  to  coordinate  such  changes  is 
necessary  in  order  to  make  sure  all  potentially  affected  parties  are  properly  consulted.  Without  such 
a  mechanism,  changes  can  be  made  without  such  consultation  or  perhaps  without  being  properly 
reflected  in  the  WBS. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE; 

(COMPLETELY  AGREE  =  6, 5, 4, 3,2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(OS)  -  005 

QUESTION;  The  procurement  personnel  staffing  has  had  continuity  throughout  the  software  life 
cycle  phases. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  staffing  continuity  is  determined  by  the  rate  of  turnover  of  personnel  during 
and  across  the  life  cycle  phases.  If  the  same  personnel  (or  at  least  a  reasonable  ration  of  the  same 
personnel)  are  not  available  from  phase  to  phase,  then  there  is  likely  to  be  a  perturbation  in  the 
schedule,  cost,  functional  requirements,  and  quality  of  the  deliverable  products.  Turnover  of  key 
personnel  should  be  minimal  with  no  sharp  variations.  See  AFP  800-48  for  more  information. 

GLOSSARY: 

Continuity.  Lack  of  turnover  and  sharp  change  in  personnel  staff. 

RESPONSE  INSTRUCTIONS: 

F/1:  50%  or  more  turnover  during  or  between  any  one  phase:  (Concept,  Demonstration, 

Development,  Deployment) 

E/2:  40%  to  50%  turnover  during  or  between  any  one  phase 

D/3:  30%  to  40%  turnover  during  or  between  any  one  phase 

C/4:  20%  to  30%  turnover  during  or  between  any  one  phase 

B/5:  10%  to  20%  turnover  during  or  between  any  one  phase 

A/6:  0%  to  10%  turnover  during  or  between  all  phases 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  -  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(OS)  -  006 

QUESTION:  The  ratio  of  experienced  procurement  project  personnel  to  the  total  number  of  project 
personnel  has  been  adequate  throughout  the  software  life  cycle  phases. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  One  key  to  developing  and  supporting  software  is  to  have  experienced  personnel, 
especially  in  the  key  leadership  positions.  Experience  with  the  subject  system,  similar  systems, 
technologically  similar  problems,  the  management  problems  of  similar  systems,  and  the  interfaces 
with  the  subject  system  results  in  better  managed,  higher  quality  software  products.  Experienced 
personnel  also  shorten  the  learning  curve  for  less  experienced  personnel.  See  AFP  800-48  for  more 
information. 

GLOSSARY: 

Experienced  Personnel.  Personnel  who  have  an  extensive  historical  perspective  of  the  subject 
system  and  its  software  requirements,  as  well  as  the  technical  expertise  and/or  management 
expertise  required  to  efficiently  implement  the  required  software  solutions. 

RESPONSE  INSTRUCTIONS: 

F/1;  10%  or  less  experienced  personnel  during  any  one  phase  (Concept,  Demonstration, 

Development,  Deployment) 

E/2:  10%  to  20%  experienced  personnel  during  any  one  phase 

D/3:  20%  to  30%  experienced  personnel  during  any  one  phase 

C/4:  30%  to  40%  experienced  personnel  during  any  one  phase 

B/5:  40%  to  50%  experienced  personnel  during  any  one  phase 

A/6:  50%  or  more  experienced  personnel  during  any  one  phase 

RESPONSE  RATIONALE: 


RESPONSE  SCOREi 

(COMPLETELY  AGREE  =  6,5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(OS)  -  007 

QUESTION;  The  number  of  procurement  personnel  has  been  adequate  throughout  the  software  life 
cycle  phases. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  number  of  personnel  should  be  sufficient  to  support  the  responsibilities 
required  by  the  program.  Sufficient  number  is  dependent  on  matching  the  experience,  workload,  and 
productivity  requirements.  Tjrpically,  fewer  persons  are  required  during  the  concept  phase,  more 
persons  during  the  demonstration  and  then  development  phases,  and  gradually  fewer  persons  prior 
to  the  deployment  phase  of  the  project.  One  method  of  determining  "sufficient"  is  to  assume  the 
allocated  number  of  personnel  is  the  most  optimum,  and  determine  the  ratio  of  actual  to  allocated 
personnel.  Guidelines  for  an  evaluation  response  based  on  this  ratio  are  given  below.  See  AFP 
800-48  for  more  information. 

GLOSSARY: 

Number  of  Procurement  Personnel.  The  count  of  personnel  directly  responsible  for  procurement 
functions  relative  to  the  subject  system.  This  includes  direct  software  project  management,  technical 
staff,  and  support  staff.  Only  those  staff  positions  directly  allocated,  or  through  direct  assignment  of 
an  allocated  position,  should  be  considered.  If  necessary,  a  "full  time  equivalent"  (e.g.,  part  of  a 
position  that  is  shared  among  one  or  more  other  system)  value  can  be  used. 

RESPONSE  INSTRUCTIONS: 

F/1:  0%  to  50%  of  allocated  load  during  any  life  cycle  phase  (Concept,  Demonstration, 

Development,  Deployment) 

E/2:  50%  to  60%  of  allocated  load  during  any  life  cycle  phase 

D/3:  60%  to  70%  of  allocated  load  during  any  life  cycle  phase 

C/4:  70%  to  80%  of  allocated  load  during  any  life  cycle  phase 

B/5:  80%  to  90%  of  allocated  load  during  any  life  cycle  phase 

A/6:  90%  or  more  of  allocated  load  during  all  life  cycle  phase 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(OS)  -  008 

QUESTION:  The  development  contractor  personnel  staffing  has  had  continuity  throughout  the 
software  life  cycle  phases. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  staffing  continuity  is  determined  by  the  rate  of  turnover  of  personnel  during 
and  across  the  life  cycle  phases.  If  the  same  personnel  (or  at  least  a  reasonable  ratio  of  the  same 
personnel)  are  not  available  from  phase  to  phase,  then  there  is  likely  to  be  a  perturbation  in  the 
schedule,  cost,  functional  requirements,  and  quality  of  the  deliverable  products.  Turnover  of  key 
personnel  should  be  minimal  with  no  sharp  variations.  See  AFP  800-48  for  more  information. 

GLOSSARY: 

Continuity.  Lack  of  turnover  and  sharp  change  in  personnel  staff. 

RESPONSE  INSTRUCTIONS: 

F/1:  50%  or  more  turnover  during  or  between  any  one  phase  (Concept,  Demonstration, 

Development,  Deployment) 

E/2:  40%  to  50%  turnover  during  or  between  any  one  phase 

D/3:  30%  to  40%  turnover  during  or  between  any  one  phase 

C/4:  20%  to  30%  turnover  during  or  between  any  one  phase 

B/5:  10%  to  20%  turnover  during  or  between  any  one  phase 

A/6:  0%  to  10%  turnover  during  or  between  all  phases 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(OS)  -  009 

QUESTION:  The  ratio  of  experienced  development  contractor  project  personnel  to  the  total  number 
of  project  personnel  has  been  adequate  throughout  the  software  life  cycle  phases. 

ACTIVITY(IES);  Development  Contractor 

EXPLANATION:  One  key  to  developing  and  supporting  software  is  to  have  experienced  personnel, 
especially  in  the  key  leadership  positions.  Experience  with  the  subject  system,  similar  systems, 
technologically  similar  problems,  the  management  problems  of  similar  systems,  and  the  interfaces 
with  the  subject  system  results  in  better  managed,  higher  quality  software  products.  Experienced 
personnel  also  shorten  the  learning  curve  for  less  experienced  personnel.  See  AFP  800-48  for  more 
information. 

GLOSSARY: 

Experienced  Personnel.  Personnel  who  have  an  extensive  historical  perspective  of  the  subject 
system  and  its  software  requirements,  as  well  as  the  technical  expertise  and/or  management 
expertise  required  to  efficiently  implement  the  required  software  solutions. 

RESPONSE  INSTRUCTIONS: 

F/1:  10%  or  less  experienced  personnel  during  any  one  phase  (Concept,  Demonstration, 

Development,  Deployment) 

E/2:  10%  to  20%  experienced  personnel  during  any  one  phase 

D/3:  20%  to  30%  experienced  personnel  during  any  one  phase 

C/4:  30%  to  40%  experienced  personnel  during  any  one  phase 

B/5:  40%  to  50%  experienced  personnel  during  any  one  phase 

A/6:  50%  or  more  experienced  personnel  during  all  phases 

RESPONSE  RATIONALE: 


RESPONSE  SCORE; 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(OS)  -  010 

QUESTION:  The  number  of  development  contractor  personnel  has  been  adequate  throughout  the 
software  life  cycle  phases. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  number  of  personnel  should  be  sufficient  to  support  the  responsibilities 
required  by  the  program.  Sufficient  number  is  dependent  on  matching  the  experience,  workload,  and 
productivity  requirements.  Typically,  fewer  persons  are  required  during  the  early  requirements 
phase,  more  persons  during  the  design  and  implementation  phases,  and  gradually  fewer  persons 
prior  to  the  production  phase  of  the  project.  There  are  few  good  guidelines  as  to  what  is  "sufficient" 
for  a  development  contractor.  The  use  of  cost  estimation  equations  as  proposed  in  the  literature  (e.g., 
B.  Boehm's  COCOMO  model)  is  sharp  increases  or  decreases  in  the  number  of  personnel  do  not 
occur.  See  AFP  800-48  for  more  information. 

GLOSSARY: 

Number  of  Development  Contractor  Personnel.  The  count  of  personnel  directly  responsible  for 
development  contractor  functions  relative  to  the  subject  system.  This  includes  direct  software 
project  management,  technical  staff,  and  support  staff.  Only  those  staff  positions  directly  assigned 
should  be  considered.  If  necessary,  a  "full  time  equivalent"  (e.g.,  part  of  a  position  that  is  shared 
among  one  or  more  other  systems)  value  can  be  used. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE' 

(COMPLETELY  AGREE  =  6, 5,4,3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(OS)  -  Oil 

QUESTION;  The  operational  support  personnel  staffing  has  had  continuity  throughout  the  soft¬ 
ware  life  cycle  phases. 

ACTIVITY(IES):  Operational  Support 

EXPLANATION:  The  staffing  continuity  is  determined  by  the  rate  of  turnover  of  personnel  during 
and  across  the  life  cycle  phases.  If  the  same  personnel  (or  at  least  a  reasonable  ratio  of  the  same 
personnel)  are  not  available  from  phase  to  phase,  then  there  is  likely  to  be  a  perturbation  in  the 
schedule,  cost,  functional  requirements,  and  quality  of  the  deliverable  products.  Turnover  of  key 
personnel  should  be  minimal  with  no  sharp  variations.  See  AFP  800-48  for  more  information. 

GLOSSARY: 

Continuity.  Lack  of  turnover  and  sharp  change  in  personnel  staff. 

RESPONSE  INSTRUCTIONS: 

F/1:  50%  or  more  turnover  during  or  between  any  one  phase  (Concept,  Demonstration, 

Development,  Deployment) 

E/2:  40%  to  50%  turnover  during  or  between  any  one  phase 

D/3:  30%  to  40%  turnover  during  or  between  any  one  phase 

C/4:  20%  to  30%  turnover  during  or  between  any  one  phase 

B/5:  10%  to  20%  turnover  during  or  between  any  one  phase 

A/6:  0%  to  10%  turnover  during  or  between  all  phases 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(OS)  -  012 

QUESTION:  The  ratio  of  experienced  operational  support  project  personnel  to  the  total  number  of 
project  personnel  has  been  adequate  throughout  the  software  life  cycle  phases. 

ACTIVITY(IES):  Operational  Support 

EXPLANATION:  One  key  to  supporting  software  is  to  have  experienced  personnel,  especially  in  the 
key  leadership  positions.  Experience  with  the  subject  system,  similar  systems,  technologically 
similar  problems,  the  management  problems  of  similar  systems,  and  the  interfaces  with  the  subject 
system  results  in  better  managed,  higher  quality  software  revisions.  Experienced  support  personnel 
also  shorten  the  learning  curve  for  less  experienced  support  personnel.  See  AFP  800-48  for  more 
information. 

GLOSSARY: 

Experienced  Personnel.  Personnel  who  have  an  extensive  historical  perspective  of  the  subject 
system  and  its  software  requirements,  as  well  as  the  technical  expertise  and/or  management 
expertise  required  to  efficiently  implement  modifications  to  the  delivered  software  products. 

RESPONSE  INSTRUCTIONS: 

F/1:  10%  or  less  experienced  personnel  during  any  one  phase  (Concept,  Demonstration, 

Development,  Deployment) 

E/2;  10%  to  20%  experienced  personnel  during  any  one  phase 

D/3:  20%  to  30%  experienced  personnel  during  any  one  phase 

C/4:  30%  to  40%  experienced  personnel  during  any  one  phase 

B/5:  40%  to  50%  experienced  personnel  during  any  one  phase 

A/6:  50%  or  more  experienced  personnel  during  all  phases 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3,2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(OS)  -  013 

QUESTION:  The  number  of  operational  support  personnel  has  been  adequate  throughout  the 
software  life  cycle  phases. 

ACTIVITY(IES):  Operational  Support 

EXPLANATION:  The  number  of  personnel  should  be  sufficient  to  support  the  responsibilities 
required  by  the  program.  Sufficient  number  is  dependent  on  matching  the  experience,  workload,  and 
productivity  requirements.  Typically,  fewer  operational  support  persons  are  required  during  the 
concept  phase,  more  persons  during  the  demonstration  and  then  development  phases,  and  maximum 
personnel  during  the  deployment  phase  of  the  project.  One  method  of  determining  "sufficient"  is  to 
assume  the  allocated  number  of  personnel  is  the  most  optimum  and  determine  the  ratio  of  actual  to 
allocated  personnel.  Guidelines  for  an  evaluation  response  based  on  this  ratio  are  given  below.  See 
AFP  800-48  for  more  information. 

GLOSSARY: 

Number  of  Operational  Support  Personnel.  The  count  of  personnel  directly  responsible  for 
operational  support  functions  relative  to  the  subject  system.  This  includes  direct  software  project 
management,  technical  staff,  and  support  staff.  Only  those  staff  positions  directly  allocated,  or 
through  direct  assignment  of  an  allocated  position,  should  be  considered.  If  necessary,  a  "full  time 
equivalent"  (e.g.,  part  of  a  position  that  is  shared  among  one  or  more  other  systems)  value  can  be 
used. 

RESPONSE  INSTRUCTIONS: 

F/1:  0%  to  50%  of  allocated  load  during  any  life  cycle  phase  (Concept,  Demonstration, 

Development,  Deployment) 

E/2:  50%  to  60%  of  allocated  load  during  any  life  cycle  phase 

D/3:  60%  to  70%  of  allocated  load  during  any  life  cycle  phase 

C/4:  70%  to  80%  of  allocated  load  during  any  life  cycle  phase 

B/5:  80%  to  90%  of  allocated  load  during  any  life  cycle  phase 

A/6:  90%  or  more  of  allocated  load  during  all  life  cycle  phase 

RESPONSE  RATIONALE: 


RESPONSE  SCORE; 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(OS)  -  014 

QUESTION:  The  internal  interfaces  among  procurement  organization  elements  have  been 

adequate. 

ACTIVITY(IES);  Procurement 

EXPLANATION:  Internal  organization  elements  might  include  the  program  office,  configuration 
management  organization,  development  test  and  evaluation  agency,  operational  test  and  evaluation 
agency,  independent  verification  and  validation  organization,  and  various  interservice  elements  as 
appropriate.  Characteristics  of  the  interfaces  to  be  assessed  include  proper  decision  process  informa¬ 
tion  flow,  effectiveness  of  information  flow,  and  adherence  to  regulations  and  guidelines  for  interface 
responsibility. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(OS)  -  015 

QUESTION:  The  internal  interfaces  among  development  contractor  organization  elements  have 
been  adequate. 

ACTrVITY(IES):  Development  Contractor 

EXPLANATION:  Internal  organization  elements  might  include  the  project  management, 

configuration  management  group,  project  technical  staff,  hardware  and  software  group,  quality 
assurance  group,  independent  test  group,  contract  management,  and  corporate  management. 
Characteristics  of  the  interfaces  to  be  assessed  include  proper  decision  process  information  flow, 
effectiveness  of  information  flow,  and  adherence  to  regulations  and  guidelines  for  interface 
responsibility. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6,5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 

Question  Number  SPM(OS)  -  016 

QUESTION:  The  internal  interfaces  among  operational  support  organization  elements  have  been 
adequate. 

ACTIVITY(IES):  Operational  Support 

EXPLANATION:  Internal  organization  elements  might  include  the  project  management, 

configuration  management  group,  project  technical  staff,  hardware  and  software  groups,  quality 
assurance  group,  independent  test  group,  contract  management,  and  supporting  and  using  command 
management.  Characteristics  of  the  interfaces  to  he  assessed  include  proper  decision  process 
information  flow,  efifectiveness  of  information  flow,  and  adherence  to  regulations  and  guidelines  tor 
interface  responsibility. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1 


=  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(OS)  -  017 

QUESTION:  The  procurement  physical  organization  structure  has  been  adequate. 

ACTrVlTY(IES):  Procurement 

EXPLANATION:  The  project  physical  structure  is  typically  represented  in  an. organization  chart. 
The  physical  organization  should  have  a  logical  relationship  to  the  project  WBS,  with  the  specific 
physical  organization  elements  having  a  well-defined  functional  responsibility  for  a  part  of  the  WBS. 
The  software  parts  of  the  physical  structure  should  be  clearly  identified  and  adequate  to  accomplish 
the  responsibilities  inherent  in  the  WBS. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

F/1:  No  physical  organization  chart  or  equivalent  is  available,  or  it  is  not  current. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(OS)  -  018 

QUESTION:  The  development  contractor  physical  organization  structure  has  been  adequate. 
ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  project  physical  structure  is  typically  represented  in  an  organization  chart. 
The  physical  organization  should  have  a  logical  relationship  to  the  project  WBS,  with  the  specific 
physical  organization  elements  having  a  well-defined  functional  responsibility  for  a  part  of  the  WBS. 
The  software  parts  of  the  physical  structure  should  be  clearly  identified  and  adequate  to  accomplish 
the  responsibilities  inherent  in  the  WBS. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

F/1:  No  physical  organization  chart  or  equivalent  is  available,  or  it  is  not  current. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(OS)  -  019 

QUESTION:  The  operational  support  physical  organization  structure  has  been  adequate. 
ACTIVITY(IES):  Operational  Support 

EXPLANATION:  The  project  physical  structure  is  t5rpically  represented  in  an  organization  chart. 
The  physical  organization  should  have  a  logical  relationship  to  the  project  WBS,  with  the  specific 
physical  organization  elements  having  a  well-defined  functional  responsibility  for  a  part  of  the  WBS. 
The  software  parts  of  the  physical  structure  should  be  clearly  identified  and  adequate  to  accomplish 
the  responsibilities  inherent  in  the  WBS. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

F/1:  No  physical  organization  chart  or  equivalent  is  available,  or  it  is  not  current. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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SOFTWARE  PROJECT  MANAGEMENT  DESIGN  METHODS 

The  questions  SPM(DM)-001  through  SPM(DM)-018  address  adequacy  of  software  project 
management  design  methods  for  the  procurement,  development  contractor,  and  operational  support 
activities.  Project  management  design  methods  are  established  so  that  the  functional  requirements 
of  a  project  can  be  more  efficiently  transcribed  into  an  implemented  product.  The  design  methods 
include  standards,  conventions,  regulations,  directives,  software  language,  review  methods,  high- 
level  representation  techniques  and  methodologies,  automated  design  aids,  and  so  forth. 

There  are  generally  two  aspects  of  a  design  method  that  need  to  be  evaluated.  First,  does  the 
design  method  facilitate  production  of  high  quality  software  within  limited  available  resources? 
Second,  can  the  design  method  be  transitioned  to  the  software  support  activity  for  the  evolution  of 
the  software  products  during  postdeployment  support? 

The  procurement  activity  is  responsible  for  ensuring  that  adequate  design  methods  have  been 
required  through  the  PMP,  CDRLs,  SOW,  and  any  other  procurement  documents.  The  procurement 
activity  is  also  responsible  for  understanding  the  nature  and  importance  of  the  development 
contractor  design  methods  and  the  effects  that  software  design  tradeoffs  might  have  on  the  effective 
implementation  of  the  desired  system.  The  procurement  activity  should  have  its  own  methods  of 
assessing  whether  system  requirements  and  design  specifications  of  those  requirements  are 
consistent,  and  whether  the  design  adequately  implements  the  requirements. 

The  development  contractor  activity  is  responsible  for  establishing  design  standards  appropriate 
for  the  software  being  developed  so  that  an  operationally  effective  and  supportable  system  is 
produced.  Design  representation  techniques  and  automated  design  aids  should  be  appropriate  for 
development  of  the  software  design  in  an  efficient  manner  and  for  transition  to  the  operational 
support  activity. 

The  operational  support  activity  is  responsible  for  making  sure  the  development  contractor  is 
required  to  use  design  methods  that  can  be  effectively  used  during  postdeployment  support.  Such 
design  methods  may  require  training  for  the  support  activity  personnel.  The  operational  support 
activity  also  has  the  responsibility  to  ensure  that  the  support  environment  is  to  be  supplied  with  all 
the  necessary  design  aids  to  effectively  accomplish  software  support. 
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QUESTION  DATA  SHEET 


Question  Number  SPM(DM)  -  001 

QUESTION:  The  procurement  design  analysis  studies  have  provided  adequate  design  guidelines  for 
the  development  contractor. 

ACTIVITY(IES):  Procurement 

EXPLANATION;  Design  analysis  studies  include  feasibility  studies  on  the  use  of  computer 
resources,  tradeoff  studies  concerning  programming  language  and  instruction  set  architecture 
selections,  alternate  approaches  for  implementing  security  requirements,  alternate  approaches  for 
achieving  operational  interoperability,  and  investigations  of  support  concepts  and  environments. 
These  studies  are  usually  part  of  the  Concept  Exploration  and  Demonstration  and  Validation  life 
cycle  phases.  The  design  guidelines  that  result  range  from  specification  of  language  and  operational 
computer  of  choice  to  a  working  prototype  of  the  complete  system. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE; 


RESPONSE  SCORE; 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(DM)  -  002 
QUESTION:  The  standards  for  software  design  required  by  the  procurement  activity  are  adequate. 
ACTIVITY(IES):  Procurement 

EXPLANATION:  The  standards  minimally  include  development  contractor  software  development 
regulations  such  as  DoD-STD-2167A,  DoD-STD-2168,  MIL-STD-483A,  MIL-STD-490A,  and  MIL- 
STD-1521B.  Generally,  the  RFP/CDRL/SOW  will  indicate  minimal  software  design  standards,  and 
the  related  Data  Item  Descriptions  will  indicate  the  format  and  content  of  the  resulting  design 
specifications. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE; 

(COMPLETELY  AGREE  =  6,5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(DM)  -  003 

QUESTION:  The  software  design  methodology  used  by  the  development  contractor  is  adequate. 
ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  Typical  design  methodologies  include  approaches  for  life  cycle  events,  personnel 
allocation  by  function,  support  resource  management,  and  schedule/budget  analyses.  The  life  cycle 
approach  might  be  top  down,  iterative  refinement,  waterfall,  prototype,  or  some  combination.  The 
personnel  allocation  by  function  method  may  prescribe  use  of  design-only,  code-only,  integrate-only, 
test-only,  or  management-only  personnel.  Or,  it  may  prescribe  personnel  groups  that  handle  some 
combination  of  these  functions.  Support  resource  management  could  include  specification  of  the 
resources  such  as  requirements  analysis  tool,  structured  analysis  design  tool,  and  automated  tools 
for  specification  generation  and  translation  to  PDL  and  implemented  source  code.  Simulators  for 
design  specifications  and  use  of  formal  verification  languages  are  other  possible  aspects  of  design 
methodology.  The  use  of  techniques  such  as  hierarchy  diagrams,  HIPOs,  N  by  N  Charts,  and  data 
flow  diagrams  is  important  for  representing  the  design  and  is  the  foundation  of  specific  design 
methods. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4,3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(DM)  -  004 

QUESTION:  The  design  standards  and  methods  adopted  for  use  by  the  operational  support  activity 
during  postdeployment  software  support  are  adequate. 

ACTIVITY(IES):  Operational  Support 

EXPLANATION:  The  operational  support  activity  should  adopt  design  standards  and  methods 
consistent  with  internal  site  product  support  standards  and  also  consistent  with  the  design 
standards  and  methods  used  by  the  development  contractor.  The  CRLCMP  should  include 
information  concerning  the  design  methods  selected. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE" 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(DM)  -  005 

QUESTION:  The  System  Design  Review  process  has  been  adequate. 

ACTIVITY(IES):  Procurement  and  Operational  Support 

EXPLANATION:  The  objective  of  the  System  Design  Review  (SDR)  is  to  formally  assess  the 
allocated  system  requirements  before  proceeding  with  the  preliminary  design  of  the  computer 
hardware  and  software  configuration  items.  The  SDR  should  include  review  of  the  detailed  system- 
level  design  and  the  allocation  of  system  functions  to  individual  hardware  and  software  configuration 
items.  The  SDR  should  include  evaluation  of  the  optimization,  traceability,  completeness,  and  risks 
associated  with  the  allocated  technical  requirements.  A  successful  SDR  will  be  predicated  on  the 
determination  that  the  System  Specification  is  an  adequate  basis  for  developing  computer  hardware 
and  software  configuration  items. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(DM)  -  006 


QUESTION:  The  software  requirements  appear  to  be  reasonable, 

ACTrVlTY(IES):  Procurement  and  Operational  Support 

EXPLANATION:  The  software  requirements  are  initially  derived  at  the  functional  level  from 
procurement  documents  such  as  the  Operational  Requirement  Document,  Program  Management 
Directive,  Program  Management  Plan,  and  eventually  the  System/Segment  Specification  (A-spec). 
The  software  requirements  must  be  an  integral,  well-defined  part  of  the  overall  system 
requirements,  and  it  must  be  clear  what  the  relationship  is  among  the  software  requirements  and 
the  system  mission  functions.  Whether  the  software  requirements  are  reasonable  depends  on  the 
total  number  of  requirements,  the  technology  necessary  to  implement  the  requirements  and 
associated  functionality  in  software,  the  schedule  and  budget  for  development  and  support,  and  the 
environmental  considerations  of  software  personnel  skills,  interface  requirements  to  the  system, 
parallel  hardware/software  development  requirements,  and  the  system's  functional  mission. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(DM)  -  007 

QUESTION:  The  number  of  software  requirements  that  cannot  be  traced  to  an  end  item  product  is 
minimal. 

ACTIVITY(IES):  All 

EXPLANATION:  Procurement  is  responsible  for  the  initial  identification  and  partial  allocation  of 
software  requirements  to  functional  end  item  components  in  the  System/Segment  Specification 
(Functional  Baseline).  The  operational  support  activity  has  a  responsibility  to  assist  in  the  definition 
of  this  Functional  responsibility  to  assist  in  the  definition  of  this  Functional  Baseline  from  the  user 
and  supporter  viewpoints.  The  development  contractor  activity  is  responsible  for  continuing  the 
software  allocation  process  from  the  high-level  functional  description  completely  through  to  the  low- 
level  software  modules  and  routines.  It  should  be  possible  to  trace  each  software  requirement  all  the 
way  down  to  the  set  of  modules  and  routines  that  implement  the  requirement  and  to  the  specific  test 
suite  that  verifies  and  validates  the  requirement.  All  requirements  should  be  traceable  to  the  CSCI 
level  at  the  conclusion  of  preliminary  design  phase.  See  AFP  800-48  for  more  information. 

GLOSSARY: 

End  Item.  An  implemented  unit  that  is  not  decomposed  further  for  purposes  of  identification. 
For  procurement  purposes,  the  usual  end  item  is  a  CSCI.  For  development  contractor  purposes  (and 
related  unit,  component,  and  integration  testing),  the  end  item  may  be  the  routine.  The  end  item  to 
consider  will  also  depend  on  the  software  development  phase. 

RESPONSE  INSTRUCTIONS: 

General  Guideline.  For  some  systems  90%  requirements  traceability  will  be  a  low  risk;  for  other 
systems  it  may  be  a  high  risk.  Fuzzy  indicators  are  if  15%  to  20%  of  the  requirements  are  not 
traceable,  then  the  software  development  is  a  medium  to  high  risk  of  compromising  the  desired 
mission  goals.  A  more  specific  set  of  guidelines  is: 

F/1:  0%  to  50%  traceability  to  an  end  item  appropriate  to  phase 

E/2:  50%  to  60%  traceability  to  an  end  item  appropriate  to  phase 

D/3:  60%  to  70%  traceability  to  an  end  item  appropriate  to  phase 

C/4:  70%  to  80%  traceability  to  an  end  item  appropriate  to  phase 

B/5:  80%  to  90%  traceability  to  an  end  item  appropriate  to  phase 

A/6:  90%  to  100%  traceability  to  an  end  item  appropriate  to  phase 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(DM)  -  008 

QUESTION:  The  number  of  software  requirements  that  cannot  he  tested  are  minimal. 
ACTIVITY(IES):  All 

EXPLANATION:  In  the  derivation  of  requirements,  it  is  the  responsibility  of  all  activities  to 
specify,  at  the  appropriate  level  of  specification,  software  requirements  in  a  way  such  that  tests  can 
be  defined  to  verify  and  validate  that  the  software  requirements  have  been  met.  The  requirements 
may  include  a  tolerance  range  of  possible  outcomes,  a  minimum/maximum  absolute  value,  a 
subjective  rating  of  a  feature,  or  a  domain/range  of  hardware  and  software  outcomes.  Although  it  is 
not  necessary  to  specify  test  criteria  to  know  whether  a  software  requirement  is  testable,  it  is  the 
best  way  to  make  sure  there  is  no  misinterpretation  of  whether  a  software  requirement  has  passed  or 
failed  a  test.  See  AFP  800-48  for  more  information. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

General  Guideline.  For  some  systems  90%  requirements  testability  will  be  a  low  risk,  for  other 
systems  it  may  be  a  high  risk.  Fuzzy  indicators  are  if  15%  to  20%  of  the  requirements  are  not 
testable,  then  the  software  development  is  a  medium  to  high  risk  of  compromising  the  desired 
mission  goals.  A  more  specific  set  of  guidelines  is: 

F/1:  0%  to  50%  requirements  have  testable  specifications 

E/2:  50%  to  60%  requirements  have  testable  specifications 

D/3:  60%  to  70%  requirements  have  testable  specifications 

C/4:  70%  to  80%  requirements  have  testable  specifications 

B/5:  80%  to  90%  requirements  have  testable  specifications 

A/6:  90%  to  100%  requirements  have  testable  specifications 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(DM)  -  009 

QUESTION:  The  profile  of  changes  to  software  requirements  is  reasonable. 

ACTIVITY(IES):  Procurement  and  Development  Contractor 

EXPLANATION:  The  number  of  change  actions  (e.g.,  ECPs)  that  impact  the  software  require¬ 
ments,  the  severity/criticality  of  the  changes,  and  the  number  of  such  changes  opened/closed  over  a 
given  time  period  determine  what  the  change  profile  is.  This  profile  will  generally  have  a  higher 
number  of  changes  early  in  the  development,  decreasing  with  occasional  upward  spikes,  to  very  few 
changes  near  the  end  of  development.  Too  many  changes  that  impact  requirements,  changes  that 
are  extremely  severe,  changes  that  are  open  for  a  long  time,  and  erratic  increases  and  decreases  in 
the  unresolved  actions  are  indicative  of  potential  problems.  Change  requests  result  from  action 
items  that  are  derived  during  informal  and  formal  project  reviews.  See  AFP  800-48  for  more 
information. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(DM)  -  010 

QUESTION:  The  profile  of  unresolved  software  review  action  items  is  reasonable. 

ACTrVlTY(IES):  Procurement  and  Operational  Support 

EXPLANATION:  Unresolved  (open)  software  review  action  items  result  from  informal  and  formal 
software  design  reviews.  Unresolved  action  items  are  expected  to  spike  upward  at  each  review  and 
then  exhibit  exponentially  decreasing  behavior.  Programs  that  issue  clearly  written  specifications 
will  experience  spikes  that  are  lower.  Programs  with  good  communication  will  have  a  higher  rate  of 
exponential  decay.  The  count  of  unresolved  software  review  action  items  must  be  maintained  by  the 
program  office  as  well  as  the  development  contractor.  See  AFP  800-48  for  more  information. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(DM)  -  Oil 

QUESTION:  The  development  contractor  requirements  analysis  process  has  been  adequate. 
ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  A  complete  set  of  functional  and  performance  requirements  must  be  established 
for  each  CSCI.  The  requirements  analysis  accomplished  during  the  Demonstration  and  Validation 
phase,  and  subsequent  requirements  validation,  must  continue  during  the  Engineering  and  Manu¬ 
facture  Development  phase  to  completely  define  the  requirements.  Interface  requirements  must  be 
defined  between  CSCIs  and  HWCIs.  All  adaptations  needed  to  accommodate  different  user  sites 
must  be  identified.  Requirements  analysis  must  evaluate  requirements  for  completeness, 
consistency,  adequacy,  testability,  understandability,  and  supportability.  As  mission  needs  change, 
additional  analyses  may  be  required  to  determine  the  impact  on  software  requirements. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(DM)  -  012 

QUESTION:  The  development  contractor  top-level  design  process  has  been  adequate. 
ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  A  modular  top-level  software  design  should  be  developed  from  the  software 
requirements.  The  preliminary  design  process  should  consider  various  design  alternatives, 
analytical  results,  tradeoff  studies,  and  capability  to  accommodate  change.  The  design  should 
identify  computer  software  components  (CSC)  and  define  the  data  interfaces,  control  flow,  and 
resource  budgets  for  memory  and  execution  time  at  the  CSC  level.  Functional  software 
requirements  should  be  assigned  to  CSCs  of  the  top-level  design.  An  initial  data  base  design  should 
define  structure  and  organization  of  the  data  base.  The  design  of  formal  and  informal  tests  should  be 
developed  and  documented  in  the  software  plan  for  testing  compliance  of  each  CSCI  with  each 
applicable  software  and  interface  requirement.  The  preliminary  design  process  culminates  with  the 
Preliminary  Design  Review  conducted  by  both  procurement  and  the  development  contractor 
activities  and  monitored  by  the  operational  support  activity. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(DM)  -  013 

QUESTION:  The  development  contractor  detailed  design  process  has  been  adequate. 
ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  detailed  software  design  process  should  refine  the  CSCs  of  the  top-level 
software  design  to  successively  lower-level  design  elements  \mtil,  at  the  lowest  level,  they  specify 
individual  units  to  be  developed.  The  detailed  design  should  define  all  information  required  for 
coding  these  units,  including  control  logic,  algorithms,  data,  accuracy,  and  timing.  For  any  interfaces 
with  other  CSCIs  or  HWCIs,  detailed  interface  design  should  precisely  define  data  formats,  data 
flow,  and  timing  constraints  in  sufficient  detail  for  coding  data  structures  and  control  routines.  The 
data  base  design  should  be  defined,  including  its  constituent  items,  fields,  records,  and  files.  The 
detailed  software  test  design  should  define  the  methods  and  criteria  of  conducting  the  individual 
tests  previously  identified  in  the  Software  Test  Plan.  Each  test  case  should  be  designed  in  terms  of 
inputs,  expected  results,  and  evaluation  criteria  (e.g.,  pass/fail).  The  test  descriptions  form  the  basis 
for  subsequent  development  of  test  procedures.  Descriptions  of  formal  tests  should  require 
procurement  activity  approval.  The  detailed  design  process  culminates  with  the  Critical  Design 
Review  conducted  by  both  the  procurement  and  the  development  contractor  activities  and  monitored 
by  the  operational  support  activity. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6,5,4,3,2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(DM)  -  014 

QUESTION:  The  design  completion  of  CSCIs  relative  to  the  software  life  cycle  development 
schedule  has  been  reasonable. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  rate  at  which  a  development  contractor  completes  CSCI  designs  may  vary 
depending  on  the  software  development  method  selected.  A  prototype  method  may  allow  for  some 
CSCIs  to  be  completely  coded  before  other  CSCIs  are  even  designed.  The  design  completion  criteria 
may  thus  be  somewhat  subjective  and  should  be  tailored  to  the  particular  system.  Generally,  the 
response  guidelines  should  reflect  adequate  considerations  of  system  life  cycle  phases  and  software 
development  methodology  being  used.  See  AFP  800-48  for  more  information. 

GLOSSARY: 

Design  Completion.  CSCI  has  satisfactorily  completed  its  detailed  design  specification  (C-spec). 

Design  Deficiency.  (Number  of  CSCIs  planned  for  design  completion  minus  the  number  of 
CSCIs  actually  designed)  divided  by  the  (number  of  CSCIs  planned  for  design  completion)  all  times 
100. 

RESPONSE  INSTRUCTIONS: 

F/1:  50%  to  100%  design  deficiency 

E/2:  40%  to  50%  design  deficiency 

D/3:  30%  to  40%  design  deficiency 

C/4:  20%  to  30%  design  deficiency 

B/5:  10%  to  20%  design  deficiency 

A/6:  0%  to  10%  design  deficiency 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(DM)  -  015 

QUESTION:  The  development  contractor  monitor  of  the  subcontractor  software  design  process  has 
been  adequate. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  Any  subcontractors  to  the  prime  development  contractor  who  have  software 
development  responsibilities  should  be  required  to  apply  design  standards  and  methods  consistent 
with  the  prime  contractor's  required  standards  and  methods.  The  prime  contractor  has  ultimate 
responsibility  to  the  procurement  activity  for  delivery  of  quality  software,  hence  subcontractor  efforts 
in  this  area  must  be  carefully  monitored  and  reviewed,  much  as  the  procurement  activity  monitors 
and  reviews  the  development  contractor  software  development  effort. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

A/6:  There  are  no  subcontractors  with  software  development  responsibilities. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(DM)  -  016 

QUESTION:  The  design  specifications  for  the  software  products  contain  adequate  information  to 
implement  the  software  with  the  required  functionality  and  within  the  schedule  and  budget 
requirements. 

ACTIVITY(IES):  All 

EXPLANATION:  The  design  specifications  include  the  System/Segment  Specification,  the  top-level 
design  specification,  the  detailed  design  specifications,  interface  specifications,  data  base  design 
specifications,  and  the  test  design  specifications.  The  design  specifications  should  adequately 
capture  the  transformation  of  requirements  into  a  paper  representation  of  the  software  solution, 
sufficiently  precise  to  directly  implement  the  software. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(DM)  -  017 

QUESTION:  The  operational  support  concept  for  design  of  software  revisions  during  postdeploy¬ 
ment  software  support  is  adequate. 

ACTIVITY(IES):  Operational  Support 

EXPLANATION:  The  operational  support  concept  should  include  design  methods,  top-level  and 
detailed  design  process  approaches,  and  test  design  methods.  This  concept  should  be  consistent  with 
the  concept  used  by  the  development  contractor. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(DM)  -  018 

QUESTION;  The  operational  support  concept  for  design  review  during  postdeployment  software 
support  is  adequate. 

ACTIVITY(IES):  Operational  Support 

EXPLANATION:  The  operational  support  concept  should  include  preliminary  and  detailed  design 
reviews  as  part  of  the  software  hlock  release  process.  The  reviews  should  he  consistent  with  those 
used  during  Engineering  and  Manufacture  Development,  probably  at  a  somewhat  reduced  scope, 
proportionate  to  the  extensiveness  of  the  changes  in  the  block  release  and  how  much  the  software 
design  has  been  affected  by  the  changes. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6,5,4,3,2,1  =  COMPLETELY  DISAGREE) 
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SOFTWARE  PROJECT  MANAGEMENT  IMPLEMENTATION  METHODS 


The  questions  SPM(IM)-001  through  SPM(IM)-016  address  adequacy  of  software  project 
management  implementation  methods  for  the  procurement,  development  contractor,  and  operational 
support  activities.  Project  management  implementation  methods  are  established  so  that  the  design 
specifications  of  a  project  can  be  more  efficiently  transcribed  into  an  implemented  product.  The 
implementation  methods  include  standards,  conventions,  regulations,  directives,  software  language, 
review  methods,  low-level  representation  techniques  and  methodologies,  automated  implementation 
aids,  and  so  forth. 

There  are  generally  two  aspects  of  an  implementation  method  that  need  to  be  evaluated.  First, 
does  the  implementation  method  facilitate  production  of  high  quality  software  within  limited 
available  resources?  Second,  can  the  implementation  method  be  transitioned  to  the  software  support 
activity  for  the  evolution  of  the  software  products  during  postdeployment  support? 

The  procurement  activity  is  responsible  for  ensuring  that  adequate  implementation  methods 
(e.g.,  coding  standards,  desk  check  procedures)  have  been  required  through  the  PMP,  CDRL,  SOW, 
and  any  other  procurement  documents.  The  procurement  activity  is  also  responsible  for 
understanding  the  nature  and  importance  of  the  development  contractor  implementation  methods 
and  the  effects  that  software  implementation  tradeoffs  might  have  on  the  desired  system.  The 
procurement  activity  should  have  its  own  methods  of  assessing  whether  software  design  and  the 
implementation  of  that  design  are  consistent,  and  whether  the  implementation  is  an  adequate 
representation  of  the  design. 

The  development  contractor  activity  is  responsible  for  establishing  and  using  implementation 
standards  appropriate  for  the  software  being  developed  so  that  an  operationally  effective  and 
supportable  system  is  produced.  Implementation  representation  techniques  and  automated 
implementation  aids  should  be  appropriate  for  coding  and  integrating  the  software  in  an  efficient 
manner  and  for  transitioning  the  techniques  and  aids  to  the  operational  support  activity. 

The  operational  support  activity  is  responsible  for  making  sure  the  development  contractor  has 
requirements  to  use  implementation  methods  that  can  be  effectively  used  during  postdeployment 
support.  Such  methods  may  require  training  for  the  support  activity  personnel.  The  operational 
support  activity  also  has  the  responsibility  to  ensure  that  the  software  development  environment  is 
delivered  with  all  the  necessary  implementation  aids  to  effectively  accomplish  software  support. 
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QUESTION  DATA  SHEET 


Question  Number  SPM(IM)  -  001 

QUESTION:  The  procurement  activity  has  adequately  monitored  the  implementation  of  the 
software  design  specifications. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  time  gap  between  the  end  of  the  major  detailed  design  phase  (Critical  Design 
Review)  and  the  beginning  of  system  integration  testing  as  signaled  by  the  Test  Readiness  Review 
can  be  significant.  It  is  necessary  for  the  procurement  activity  to  carefully  monitor  development 
contractor  implementation  progress  through  status  reports,  informal  reviews,  site  visits,  interim 
demonstrations,  and  the  required  software  baseline  change  process. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(IM)  -  002 

QUESTION:  The  procurement  test  organization  interface  with  the  development  contractor  is 
adequate  enough  to  ensure  success  of  the  system  integration  tests. 

ACTIVITY(IES):  Procurement  and  Development  Contractor 

EXPLANATION:  The  primary  test  organizations  are  the  DT&E  agency,  the  OT&E  agency,  and 
possibly  an  IV&V  organization.  The  system  integration  test  success  is  directly  dependent  on  the 
implementation  progress  of  the  development  contractor.  The  operational  test  process  also  has  such 
dependency.  The  IV&V  (if  any)  will  generally  provide  for  supplementary  testing  that  provides 
greater  assurance  that  the  implementation  is  all  right  and/or  identifies  problem  areas  that  decrease 
assurance  and  lengthen  the  implementation  phase. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5,4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(IM)  -  003 

QUESTION:  The  operational  support  activity  has  been  actively  involved  with  the  development 
contractor's  software  implementation  in  order  to  learn  the  software  prior  to  officially  accepting 
software  support  responsibility. 

ACTIVITY(IES):  Operational  Support 

EXPLANATION:  All  through  the  full-scale  development,  the  operational  support  activity  should 
monitor  the  progress  of  the  development  contractor's  software  development.  During  the 
implementation  phase  (latter  part),  some  key  operational  support  activity  personnel  should  begin  to 
actively  learn  the  software  design,  implementation,  integration,  and  test.  This  may  take  the  form  of 
formal  course  training  and  hands-on  software  modification  to  informal  observations  of  the 
development  contractor  process. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

F/1:  There  are  no  plans  for  operational  support  personnel  to  actively  participate  in  the  software 

modification  process. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4,3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(IM)  -  004 

QUESTION:  The  standards  for  software  implementation  required  by  the  procurement  activity  are 
adequate. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  standards  minimally  include  development  contractor  software  development 
regulations  such  as  DoD-STD-2167A,  DoD-STD-2168,  MIL-STD-483A,  MIL-STD-490A,  and  MIL- 
STD-1521B.  Generally,  the  RFP/CDRL/SOW  will  indicate  minimal  software  implementation 
standards,  and  the  related  Data  Item  Descriptions  will  indicate  the  format  and  content  of  the 
resulting  implementation  specifications. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(IM)  -  005 

QUESTION:  The  implementation  methodology  used  by  the  development  contractor  is  adequate. 
ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  Typical  implementation  methodologies  include  approaches  for  life  cycle  events, 
personnel  allocation  by  function,  support  resource  management,  and  schedule/budget  analyses.  The 
life  cycle  approach  might  be  top  down,  iterative  refinement,  waterfall,  protot5pe,  or  some 
combination.  The  personnel  allocation  by  function  method  may  prescribe  use  of  design-only,  code- 
only,  integrate-only,  test-only,  or  management-only  personnel.  Or,  it  may  prescribe  personnel 
groups  that  handle  some  combination  of  these  functions.  Support  resource  management  could 
include  specification  of  the  resources  such  as  requirements  analysis  tool,  structured  analysis  design 
tool,  and  automated  tools  for  specification  generation  and  translation  to  PDL  and  implemented 
source  code.  Simulators  for  design  specifications  and  use  of  formal  verification  languages  are  other 
possible  aspects  of  implementation  methodology.  The  use  of  techniques  such  as  code  walkthroughs, 
simulation,  symbolic  debug,  static  code  analysis,  automated  test  case  generators,  regression  testing 
is  the  foundation  of  code/unit  test/integration  implementation  methods. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6,5, 4,3, 2,1  =  COMPLETELY  DISAGREE) 


AFOTECPAM  99-102  Volume  2  Attachment  1  1  August  1994 


97 


QUESTION  DATA  SHEET 


Question  Number  SPM(IM)  -  006 

QUESTION;  The  implementation  standards  and  methods  adopted  for  use  by  the  operational 
support  activity  during  postdeployment  software  support  are  adequate. 

ACTIVITY(IES):  Operational  Support 

EXPLANATION:  The  operational  support  activity  should  adopt  implementation  standards  and 
methods  consistent  with  internal  site  product  support  standards  and  also  consistent  with  the 
implementation  standards  and  methods  used  by  the  development  contractor.  The  CRLCMP  should 
include  information  concerning  the  implementation  methods  selected. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(IM)  -  007 

QUESTION:  The  development  contractor  monitor  of  subcontractor  software  implementation 
process  has  been  adequate. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  Any  subcontractors  to  the  prime  development  contractor  who  have  software 
development  responsibilities  should  be  required  to  apply  implementation  standards  and  methods 
consistent  with  the  prime  contractor's  required  standards  and  methods.  The  prime  contractor  has 
ultimate  responsibility  to  the  procurement  activity  for  delivery  of  quality  software,  hence 
subcontractor  efforts  in  this  area  must  be  carefully  monitored  and  reviewed,  much  as  the 
procurement  activity  monitors  and  reviews  the  development  contractor  software  development  effort. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

A/6:  There  are  no  subcontractors  with  software  development  responsibility. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(IM)  -  008 

QUESTION:  The  implementation  completion  of  CSCIs  has  been  reasonable  relative  to  the  software 
life  cycle  schedule. 

ACTrVlTY(IES):  Development  Contractor 

EXPLANATION:  The  rate  at  which  a  development  contractor  completes  CSCI  implementations 
may  vary  depending  on  the  software  development  method  selected.  A  prototype  method  may  allow 
for  some  CSCIs  to  be  completely  tested  before  other  CSCIs  are  even  designed.  The  implementation 
completion  criteria  may  thus  be  somewhat  subjective  and  should  be  tailored  to  the  particular  system. 
Generally,  the  response  guidelines  should  reflect  adequate  considerations  of  system  life  cycle  phases 
and  software  development  methodology  being  used.  See  AFP  800-48  for  more  information. 

GLOSSARY: 

Implementation  Completion.  Implementation  of  a  CSCI  is  complete  when  the  CSCI  has 
satisfactorily  completed  its  integration  testing. 

Implementation  Deficiency.  (Number  of  CSCIs  planned  for  implementation  completion  minus 
the  number  of  CSCIs  actually  implemented)  divided  by  the  (number  of  CSCIs  planned  for 
implementation  completion)  all  times  100. 

RESPONSE  INSTRUCTIONS: 

F/1:  50%  to  100%  implementation  deficiency 

E/2:  40%  to  50%  implementation  deficiency 

D/3:  30%  to  40%  implementation  deficiency 

C/4:  20%  to  30%  implementation  deficiency 

B/5:  10%  to  20%  implementation  deficiency 

A/6:  0%  to  10%  implementation  deficiency 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(IM)  -  009 

QUESTION:  The  procurement  software  project  management  support  tool  environment  is  adequate. 
ACTIVITY(IES):  Procurement 

EXPIiANATION:  During  the  Concept  Exploration  and  Demonstration  and  Validation  phases,  the 
necessary  automated  tools  and  procedures  for  procurement  project  management  (software  and 
hardware)  should  be  identified,  developed,  and/or  acquired.  The  tool  environment  should  allow  for 
budget  and  schedule  management,  mission  requirements  tracing,  product  deliverable  status 
tracking,  configuration  management  change  status  tracking,  and  management  information  com¬ 
munication  capabilities  among  participating  organizations.  A  current  list  of  required  tools,  function 
of  each  tool,  date  required,  and  date  acquired  should  be  maintained. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(IM)  -  010 

QUESTION:  The  development  contractor  software  project  management  support  tool  environment  is 
adequate. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  development  contractor  activity  is  required  to  provide  certain  management 
information  on  the  project  status  to  the  procurement  activity.  Automated  project  management 
functions  would  assist  the  development  contractor  in  this  effort.  DoD-STD-2167A  and  its  associated 
Data  Item  Descriptions,  as  called  out  in  the  CDRL,  have  specific  requirements  for  development 
contractor  data  collection  and  reporting. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

If  no  automated  project  management  tools  are  being  used,  then  the  response  should  be  less  than 

C/4. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(IM)  -  Oil 

QUESTION:  The  development  contractor  software  configuration  management  support  tool 

environment  is  adequate. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  Software  configuration  management  is  one  of  the  most  important  management 
functions  performed  by  the  development  contractor.  Frequently,  the  amount  of  information  to  be 
retained,  analyzed,  and  reported  is  very  large.  Thus,  an  automated  support  tool  environment  to  this 
process  is  essential.  Such  a  tool  must  be  supplemented  with  adequate  management  procedures. 
Such  a  tool  environment  must  have  the  capability  to  efficiently  report  on  all  software  components 
under  configuration  control,  current  and  planned  changes  to  those  components,  and  planned 
components  not  currently  under  control.  Library  management  of  software  (specification,  source, 
object,  command  language,  load  modules,  test  data,  etc.)  and  the  capability  to  automatically 
reconstruct  current  and  precious  versions  of  software  components  are  required. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

If  no  automated  project  management  tools  are  being  used,  then  the  response  should  be  less  than 

C/4. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(IM)  -  012 

QUESTION:  The  development  contractor  system  software  tool  environment  is  adequate. 
ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  precise  configuration  of  a  system  development  tool  environment  will  vary 
somewhat  depending  on  the  particular  application  and  the  complexity  of  the  software  development 
effort. 

GLOSSARY: 

System  Software  Tool.  Operating  system,  compiler,  linker  debugger,  data  base  manager, 
methodology  support  tool,  requirements  generation  tool,  host  system,  and  so  forth. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3,2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(IM)  -  013 

QUESTION:  The  development  contractor  application  test  software  tool  environment  is  adequate. 
ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  test  environment  tool  set  is  critical  for  thorough  unit  testing  and  laboratory 
integration  testing.  The  maturity  of  the  software  is  still  largely  dependent  on  how  completely  the 
software  can  be  tested. 

GLOSSARY: 

Application  Software  Test  Environment.  Software  bench,  target  machine,  laboratory  integrated 
testbed,  specialized  test  devices  and  instrumentation,  special  security  facilities,  simulators  and 
emulators,  and  so  forth. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6,5,4, 3,2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(IM)  -  014 

QUESTION:  The  operational  support  software  support  tool  environment  is  adequate. 
ACTIVITY(IES):  Operational  Support 

EXPLANATION:  The  software  support  tool  environment  is  one  of  the  critical  factors  for  software 
supportability.  The  software  revisions  must  be  developed,  configuration  controlled,  and  tested  in  a 
manner  similar  to  the  original  engineering  and  manufacture  development  effort. 

GLOSSARY: 

Software  Support  Tool  Environment.  The  systems  of  the  Software  Support  Resources.  Includes 
the  system  development  software  tool  environment  and  the  application  software  test  tool 
environment  as  required  by  the  support  environment. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE; 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(IM)  -  015 

QUESTION:  The  operational  support  software  concept  for  implementation  of  software  revisions 
during  postdeployment  software  support  is  adequate. 

ACTrVlTY(IES):  Operational  Support. 

EXPLANATION:  The  operational  support  concept  should  include  coding  style  guidelines,  methods 
for  ensuring  code  correctness  (e.g.,  quality  assurance  metrics,  code  desk  checks)  prior  to  unit  test, 
unit  and  integration  test  methods,  and  operational  test  methods.  The  concept  should  include  an 
overall  software  support  plan  that  delineates  how  software  releases  are  to  be  project  and 
configuration  managed.  This  plan  is  an  internal  detailed  specification  derived  from  the  more  high- 
level  CRLCMP. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(IM)  -  016 

QUESTION:  The  operational  support  concept  for  implementation  audits  and  reviews  during 
postdeplo3anent  software  support  is  adequate. 

ACTrVTTY(IES):  Operational  Support 

EXPLANATION:  The  Review  of  implemented  revisions  has  some  parallel  with  the  reviews  and 
audits  in  the  development  process.  However,  the  reviews  tend  to  be  less  extensive.  A  Test  Readiness 
Review  should  be  conducted  on  the  revised  software  baseline  prior  to  operational  testing  of  each 
block  release.  Informal  reviews  small-scale  functional  and  physical  configuration  audit/reviews  for 
configuration  baseline  update  integrity,  and  test  readiness  reviews  constitute  the  audits  and  reviews 
pertinent  to  the  implementation  phase  of  a  block  release. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 


108 


AFOTECPAM99  102  Volume  2  Attachment!  1  August  1994 


SOFTWARE  PROJECT  MANAGEMENT  TEST  STRATEGIES 


The  questions  SPM(TS)-001  through  SPM(TS)-020  address  adequacy  of  software  project 
management  test  strategies  for  the  procurement  development  contractor  and  operational  support 
activities.  Software  test  strategies  are  established  so  that  the  implemented  product  can  be  verified 
and  validated  against  the  requirement  specifications.  The  test  strategies  include  standards,  conven¬ 
tions,  regulations,  directives,  test  languages,  verification  and  validation  methods,  and  automated  test 
aids.  Key  to  a  reasonable  test  strategy  is  the  generation  of  a  test  plan,  test  description,  test 
procedures,  test  reports,  demonstration  tests,  configuration  management  of  test  information,  and 
transition  of  test  strategy  to  the  operational  support  activity. 

There  are  generally  two  aspects  of  a  test  strategy  that  need  to  be  evaluated.  First,  does  the  test 
strategy  facilitate  production  of  high  quality  software  within  limited  available  resources?  Second, 
can  the  test  strategy  be  used  by  the  software  support  activity  for  the  evolution  of  the  software 
products  during  postdeployment  support? 

The  procurement  activity  is  responsible  for  ensuring  that  adequate  test  methods  have  been 
required  or  are  planned  through  the  PMP,  CDRL,  SOW,  TEMP,  DT&E  and  OT&E  test  plans,  and 
any  other  procurement  documents.  The  procurement  activity  is  also  responsible  for  understanding 
the  nature  and  importance  of  the  development  contractor  test  strategies  and  the  effects  that  various 
testing  techniques  might  have  on  the  verification  and  validation  of  the  desired  system.  The 
procurement  activity  is  responsible  for  making  sure  the  DT&E,  OT&E,  IV&V,  and  development 
contractor  test  strategies  are  consistent  and  are  complementary.  The  procurement  activity  should 
have  its  own  methods  of  assessing  whether  software  requirements  have  been  adequately  verified  and 
validated  and  the  amount  and  t3rpe  of  testing  that  is  still  required  to  achieve  an  operational 
capability. 

The  development  contractor  activity  is  responsible  for  establishing  test  standards  appropriate 
for  the  software  being  developed  so  that  an  operationally  effective  and  supportable  system  is 
produced.  Test  plans,  techniques,  schedules,  and  automated  aids  should  be  appropriate  for  thorough 
test  of  the  software  implementation  and  system  integration.  The  test  techniques,  test  cases,  and  test 
environment  should  be  designed  for  transition  to  the  operational  support  activity. 

The  operational  support  activity  is  responsible  for  making  sure  the  development  contractor  is 
required  to  use  a  test  strategy  that  can  be  effectively  used  during  postdeployment  support.  The 
operational  support  activity  also  has  the  responsibility  to  ensure  that  the  postdeployment  software 
support  test  strategy  is  consistent  with  the  development  contractor's  test  strategy. 


AFOTECPAM  99-102  Volume  2  Attachment  1  1  August  1994 


109 


QUESTION  DATA  SHEET 


Question  Number  SPM(TS)  -  001 

QUESTION:  The  TEMP  adequately  describes  the  software  test  and  evaluation  process. 
ACTIVITY(IES):  Procurement  and  Operational  Support 

EXPLANATION:  The  software  test  and  evaluation  process  includes  objectives,  measures  of 
effectiveness,  organization  responsibilities  and  interfaces,  the  DT&E  and  OT&E  specific  test 
interfaces,  and  the  overall  schedule  and  funding  level  of  the  test  process.  Any  use  of  IV&V  by 
procurement  should  be  described  along  with  the  organization  relationships  and  expected  results. 
The  TEMP  is  a  concise  description  of  the  complete  system  test  process,  but  there  should  be  adequate 
attention  to  the  software  test  process  and  appropriate  reference  to  other  planning  documents  for 
more  detailed  information. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

F/1:  There  is  no  TEMP. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5,4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(TS)  -  002 

QUESTION:  The  software  test  process  for  DT&E  has  followed  the  guidelines  in  the  TEMP. 
ACTIVITYdES):  Procurement  and  Operational  Support 

EXPLANATION:  The  TEMP  reflects  top-level  input  from  the  DT&E  organization  and  should  be 
followed  if  the  TEMP  is  to  be  an  important,  high-level  planning  document.  Characteristics  to  be 
considered  include  schedule,  planning  and  utilization  of  resources,  derivation  of  high-level  measures 
of  effectiveness  for  the  specified  objectives  and  subobjectives,  and  the  communication  of  test  results 
with  other  test  organizations. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

F/1:  The  TEMP  does  not  exist  or  does  not  address  software  DT&E. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3,2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(TS)  -  003 

QUESTION:  The  software  test  process  for  OT&E  has  followed  the  guidelines  in  the  TEMP 
ACTIVITY(IES):  Proctmement  and  Operational  Support 

EXPLANATION:  The  TEMP  reflects  top-level  input  from  the  OT&E  organization  and  should  be 
followed  if  the  TEMP  is  to  be  an  important,  high-level  planning  document.  Characteristics  to  be 
considered  include  schedule,  planning  and  utilization  of  resources,  derivation  of  high-level  measures 
of  effectiveness  for  the  specified  objectives  and  subobjectives,  and  the  communication  of  test  results 
with  other  test  organizations. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

F/1:  The  TEMP  does  not  exist  or  does  not  address  software  OT&E. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(TS)  -  004 

QUESTION:  The  implementation  of  the  software  test  process  by  DT&E  and  OT&E  organizations 
has  been  adequate. 

ACTIVITY(IES):  Procurement  and  Operational  Support 

EXPLANATION:  The  TEMP  guidelines  for  the  software  test  process  should  reflect  the  DT&E  and 
OT&E  approaches.  The  actual  implementation  of  those  guidelines  is  specified  in  the  DT&E  and 
OT&E  organizations'  plans.  Check  the  effectiveness  of  the  tests  to  stress  the  software  components  in 
a  thorough  manner.  The  assurance  the  tests  provide  that  the  software  requirements  have  been 
verified  at  a  low  level  of  detail  and  validated  in  an  operationally  representative  environment  is  one 
measure  of  how  mature  the  software  is  likely  to  be  during  early  postdeployment  support. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

F/1:  The  DT&E  and  OT&E  organizations  do  not  have  any  specific  software  test  plans. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(TS)  -  005 

QUESTION:  The  test  organizations  have  incorporated  a  strategy  in  their  software  test  processes  for 
coordination  and  sharing  of  test  plans,  procedures,  and  results. 

ACTIVITY(IES):  All 

EXPLANATION:  The  test  and  evaluation  directives  require  participation  of  the  various  test 
organizations  (e.g.,  DT&E,  OT&E,  IV&V)  across  the  complete  system  life  cycle.  In  addition,  the 
requirement  is  that  these  organizations  coordinate  their  activities  so  as  to  be  more  effective  and 
thorough.  Thus,  DT&E  results  should  feed  the  OT&E  process;  the  requirements  of  the  OT&E 
process  should  affect  the  DT&E  process;  the  IV&V  process  and  results  should  not  duplicate,  but 
complement  and  supplement  the  DT&E,  OT&E,  and  development  contractor  testing  process.  The 
development  contractor  testing  process  should  be  an  integral  part  of  the  DT&E  process  and 
monitored  closely  by  the  OT&E  agency  and  operational  support  activity. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(TS)  -  006 

QUESTION:  The  requirements  for  the  development  contractor  software  test  strategy  are  clearly 
specified  in  the  RFP,  SOW,  and/or  CDRLs. 

ACTIVITY(IES):  Procurement  and  Operational  Support 

EXPLANATION:  The  best  way  to  achieve  a  good  contractor  test  strategy  is  to  require  one.  The  best 
way  to  do  this  is  through  the  RFP,  SOW,  and  the  CDRLs.  The  form,  methods,  techniques,  schedule, 
deliverables,  transition,  and  so  forth  for  the  developing  contractor  test  strategy  should  be  described 
in  the  procurement  documentation. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

F/1:  The  software  test  strategy  is  not  defined  in  any  of  the  documentation. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6,5,4,3,2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(TS)  -  007 

QUESTION:  The  use  of  an  organization  for  software  test  IV&V  support  has  been  effective. 
ACTIVITY(IES):  Procurement  and  Operation  Support 

EXPLANATION:  The  IV&V  function  is  not  always  required  or  appropriate.  For  mission  critical 
systems  with  significant  software,  it  is  usually  required.  The  software  IV&V  may  be  separate  from 
or  a  part  of  a  system  IV&V  effort.  For  IV&V  to  be  effective,  it  must  usually  be  applied  early  in  the 
software  life  cycle  and  comprise  a  significant  (e.g.,  10  to  30  percent)  portion  of  the  software 
development  cost.  There  are  specific  instances  where  IV&V  can  be  used  to  solely  support  DT&E 
and/or  OT&E  in  their  specific  functions,  in  which  case  the  IV&V  function  is  less  comprehensive  and 
costly.  This  can  be  an  especially  effective  way  to  obtain  detailed  software  stress  testing  and 
evaluation  that  might  not  be  done  because  of  a  shortage  of  test  organization  support  personnel. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

F/1:  No  IV&V  function  has  been  defined,  but  the  system  is  a  mission  critical  system  with 
significant  software  functions. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(TS)  -  008 

QUESTION:  The  overall  planning  for  software  testing  has  been  adequate. 

ACTIVITY(IES):  All 

EXPLANATION:  The  combination  of  procurement,  development  contractor,  and  operational 
support  software  test  planning  should  reflect  an  integrated  strategy  for  accomplishing  the  software 
test  process  throughout  the  software's  complete  life  cycle.  The  various  activities'  test  plans  should 
identify  and  emphasize  those  aspects  of  primary  responsibility.  In  particular,  such  plans  should 
identify  test  items,  features  to  be  tested,  features  not  to  be  tested,  relationship  of  the  test  items  to 
the  baseline  being  tested  and  the  functional  system  requirements,  test  approach  and  methodologies 
employed,  pass/fail  criteria,  specific  test  tasks  (WBS),  test  environment,  test  responsibility  and 
resource  requirements,  and  test  schedules.  The  test  plan,  test  descriptions,  test  log,  test  results, 
incidence  reports,  and  any  other  test  documentation  to  be  produced  should  be  described  in  the 
software  test  plans. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(TS)  -  009 

QUESTION:  The  software  test  approach  and  methodologies  employed  are  clearly  described  in  the 
software  test  documentation  and  appear  to  be  effective. 

ACTIVITY(IES):  All 

EXPLANATION:  The  approach  should  be  described  for  each  major  group  of  features  to  be  tested. 
The  major  activities,  techniques,  and  tools  that  are  to  be  used  should  be  described  in  enough  detail  to 
identify  major  testing  tasks  and  estimation  of  the  time  required  to  do  each  one.  The  nr>inimnm 
degree  of  completeness  should  be  specified  along  with  the  techniques  (e.g.,  path  coverage,  statement 
coverage,  domain  space  coverage)  used  to  judge  completeness.  Acceptance  criteria  constraints  such 
as  test-item  availability,  test  resource  availability,  schedule  deadlines,  and  funding  levels.  Testing 
techniques  include  code  reviews,  structure  analysis,  static  program  quality  analysis,  dynamic  path 
analysis,  coverage  analysis,  assertion  checking,  symbolic  debugging,  mutation  testing,  and  regression 
testing.  Such  techniques  can  be  applied  with  varying  success  across  the  test  phases  of  design 
verification,  unit  and  module  test,  CSCI  and  system  integration  test,  PQT/FQT,  system  test,  and 
mission  test.  Application  of  testing  techniques  across  test  phases  and  descriptions  of  how  the  overall 
test  objectives  are  to  be  achieved  should  be  clearly  specified  in  the  test  documentation. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(TS)  -  010 

QUESTION:  The  software  features  to  be  tested  and  not  to  be  tested  are  clearly  described  in  the 
software  test  documentation. 

ACTIVITY(IES):  All 

EXPLANATION:  The  software  test  documentation  should  identify  features  to  be  tested,  features 
not  to  be  tested,  and  all  combinations  of  such  features  across  the  test  cases. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

F/1:  Software  test  documentation  does  not  identify  features  to  be  tested  and  not  tested. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4,3,2, 1  =  COMPLETELY  DISAGREE) 


AFOTECPAM  99-102  Volume  2  Attachment  1  1  August  1994 


119 


QUESTION  DATA  SHEET 


Question  Number  SPM(TS)  -  Oil 

QUESTION:  The  traceability  software  features  tested/not  tested  to  the  software  functional 
requirements  are  described  in  the  software  test  documentation. 

ACTIVITY(IES):  All 

EXPLANATION:  The  software  test  documentation  should  provide  assurance  of  which  functional 
requirements  of  the  software  have/have  not  been  satisfied.  A  clear  association  of  the  functional 
requirements  with  the  software  features  and  the  related  tests  is  a  major  step  toward  providing  that 
assurance.  Typically,  a  matrix  or  written  description  is  provided.  Use  of  a  cross-reference  among 
data  dictionaries  for  requirements,  features,  and  tests  is  another  way  such  information  can  be 
presented, 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

F/1:  No  cross-reference  exists  among  software  features,  test  cases,  and  functional  requirements. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1-  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(TS)  -  012 

QUESTION:  The  software  test  deliverables  are  adequately  specified  in  the  software  test 

documentation. 

ACTIVITY(IES):  All 

EXPLANATION:  The  software  test  documentation  should  provide  a  complete  list  and  description  of 
all  software  test  deliverables.  A  software  test  plan  is  the  most  logical  location  for  such  information. 
It  should  also  be  clearly  stated  how  the  deliverables  relate  to  the  CDRLs/DIDs  in  the  case  of  the 
development  contractor. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 

Question  Number  SPM(TS)  -  013 

QUESTION:  The  software  test  criteria  used  to  determine  whether  each  test  has  passed  or  failed  are 
clearly  specified  in  the  software  test  documentation. 

ACTIVITYdES):  All 

EXPLANATION:  The  software  test  documentation  should  provide  pass/fail  criteria  for  each  test 
and  each  software  feature  to  he  tested.  The  criteria  should  be  as  objective  as  possible.  Examples  of 
criteria  include  percentage  of  statements  tested,  percentage  of  logic  paths  tested,  and  faults 
(frequency,  number  of  critical/noncritical)  allowed. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

F/1:  50%  or  more  of  the  tests/features  have  inadequate  criteria 

E/2:  40%  to  50%  of  the  tests/features  have  inadequate  criteria 

D/3:  30%  to  40%  of  the  tests/features  have  inadequate  criteria 

C/4:  20%  or  30%  of  the  tests/features  have  inadequate  criteria 

B/5:  10%  to  20%  of  the  tests/features  have  inadequate  criteria 

A/6:  0%  to  10%  of  the  tests/features  have  inadequate  criteria 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5,4,3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(TS)  -  014 

QUESTION:  The  personnel  groups  responsible  for  the  software  tests  are  adequately  identified  in 
the  software  test  documentation. 

ACTIVITY(IES);  All 

EXPLANATION:  The  software  test  documentation  should  clearly  identify  personnel  groups  who 
are  responsible  for  the  various  software  tests  across  all  the  test  phases.  In  some  cases  the  individual 
programmer  will  be  responsible;  in  other  cases  a  complete  independent  test  group  may  be 
responsible.  Responsibilities  include  managing,  designing,  preparing,  executing,  monitoring, 
checking,  resolving  anomalies,  and  acceptance/approval.  A  software  test  plan  is  the  most  likely 
source  of  this  information,  although  individual  test  case  descriptions  are  also  a  possible  source. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

F/1:  50%  or  more  tests  have  no  responsible  group  identified 

E/2:  40%  to  50%  tests  have  no  responsible  group  identified 

D/3:  30%  to  40%  tests  have  no  responsible  group  identified 

C/4:  20%  or  30%  tests  have  no  responsible  group  identified 

B/5:  10%  to  20%  tests  have  no  responsible  group  identified 

A/6:  0%  to  10%  tests  have  no  responsible  group  identified 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(TS)  -  015 

QUESTION:  The  high  risk  assumptions  of  the  software  testing  approach  along  with  contingency 
plans  for  each  such  assumption  are  adequately  described  in  the  software  test  documentation, 

ACTIVITY(IES):  All 

EXPLANATION:  The  software  test  documentation  should  clearly  identify  any  areas  of  high  risk. 
Examples  of  high  risk  tests  include  those  cases  that  would  require  interagency  (e.g.,  interservice) 
allocation  of  resources  such  as  test  ranges,  equipment,  or  personnel  for  which  the  availability  is 
scarce.  A  schedule  delay  in  such  a  test  might  cause  a  large  ripple  effect,  not  only  in  the  subject 
system  test  schedule,  but  in  the  test  schedule  of  systems  competing  for  the  same  test  resources. 
Alternate  plans  for  handling  such  difficulties  (resource  availability,  funding  level,  technical 
problems)  should  be  described.  A  software  test  plan  is  the  most  likely  source  for  identifying  the  high- 
risk  test  cases  and  high-level  contingency  plans.  The  individual  test  case  description  would  probably 
provide  more  detail  as  to  why  the  test  case  is  high  risk. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

F/1:  50%  or  more  of  the  high  risk  tests  have  no  contingency  plan,  or  high  risk  tests  exist  but  are 

not  identified 

E/2:  40%  to  50%  of  the  high  risk  tests  have  no  contingency  plan 

D/3:  30%  to  40%  of  the  high  risk  tests  have  no  contingency  plan 

C/4:  20%  or  30%  of  the  high  risk  tests  have  no  contingency  plan 

B/5:  10%  to  20%  of  the  high  risk  tests  have  no  contingency  plan 

A/6:  0%  to  10%  of  the  high  risk  tests  have  no  contingency  plan 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(TS)  -  016 

QUESTION:  The  schedule  for  software  test  milestones  is  adequately  specified  in  the  software  test 
documentation. 

ACTIVITY(IES):  Procurement  and  Development  Contractor 

EXPLANATION:  The  software  test  documentation  should  clearly  identify  the  milestones  for  each 
test.  These  milestones  are  normally  part  of  the  software  test  plan/approach,  the  software  project 
schedule  that  is  a  subpart  of  the  system  WBS  schedule,  or  the  system  development  schedule.  The 
test  milestones  and  their  relationship  to  the  overall  software  and  system  development  schedule 
should  be  specified. 

GLOSSARY: 

Test  Milestones.  Completion  of  design,  coding,  unit  test,  hardware/software  integration, 
documentation,  acceptance  are  typical  milestones. 

RESPONSE  INSTRUCTIONS: 

F/1:  50%  or  more  of  the  software  tests  have  no  milestone  schedule 

E/2:  40%  to  50%  of  the  software  tests  have  no  milestone  schedule 

D/3:  30%  to  40%  of  the  software  tests  have  no  milestone  schedule 

C/4:  20%  or  30%  of  the  software  tests  have  no  milestone  schedule 

B/5:  10%  to  20%  of  the  software  tests  have  no  milestone  schedule 

A/6:  0%  to  10%  of  the  software  tests  have  no  milestone  schedule 

RESPONSE  RATIONALE: 


RESPONSE  SCORE' 

(COMPLETELY  AGREE  =  6, 5,4, 3,2,1  =  COMPLETELY  DISAGREE) 


AFOTECPAM  99-102  Volume  2  Attachment  1  1  August  1994 


125 


QUESTION  DATA  SHEET 


Question  Number  SPM(TS)  -  017 

QUESTION:  Software  testing  is  adequately  prioritized  in  the  software  test  approach  according  to 
mission  criticality  concerns. 

ACTIVITY(IES):  All 

EXPLANATION:  The  software  test  approach  should  prioritize  the  software  testing  process 
according  to  mission  critical  features.  For  example,  if  certain  software  features  are  critical  to  the 
mission  reliability  of  the  system  or  perhaps  the  safety  of  the  system  personnel,  then  those  features 
should  receive  a  higher  priority.  Higher  priority  features  may  require  more  rigorous  tests,  more 
objective  measures  of  test  assurance,  longer  test  schedule,  and  a  more  visible  test  reporting  process. 

GLOSSARY: 

Mission  Critical  Feature.  Any  feature  of  the  system  that  will  prevent  the  completion  of  the 
mission  objective  or  impact  the  safety  of  the  personnel  who  are  part  of  the  mission  if  it  is  not 
developed  correctly. 

RESPONSE  INSTRUCTIONS: 

F/1:  No  prioritization  of  tests  is  apparent  from  the  software  documentation,  or  the  defined 
prioritization  is  not  being  followed. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3.2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(TS)  -  018 

QUESTION:  The  software  test  environment  is  adequately  identified  in  the  software  test 

documentation  and  is  adequate  for  accomplishing  the  required  testing. 

ACTIVITY(IES):  All 

EXPLANATION:  The  software  test  documentation  should  identify  the  software  test  environment 
including  host  emulation/simulation,  software  bench  testing  equipment,  integrated  laboratory 
environments,  and  operational  mission  test  environments. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

F/1:  No  identification  of  the  software  test  environment  is  documented,  or  the  environment  is 
totally  inadequate. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 


AFOTECPAM  99-102  Volume  2  Attachment  1  1  August  1994 


127 


QUESTION  DATA  SHEET 


Question  Number  SPM(TS)  -  019 

QUESTION:  The  configuration  management  of  the  software  test  process  is  adequate. 

ACTIVITY(IES):  All 

EXPLANATION:  All  activities  have  some  responsibility  for  software  test  and  evaluation.  As  such, 
each  activity  has  a  responsibility  for  its  particular  emphasis  to  maintain  appropriate  configuration 
management  of  the  test  process  and  its  documentation.  In  particular,  the  procurement  DT&E  and 
OT&E  test  plans,  approaches,  descriptions,  and  results  should  be  under  tight  configuration 
management.  Likewise,  the  development  contractor’s  software  test  documentation  should  be  a 
contract  deliverable,  perhaps  as  a  CSCI  depending  on  the  criticality  of  the  software.  The  operational 
support  software  test  documentation  should  be  carefully  controlled  throughout  the  postdeployment 
support  of  the  software. 

GLOSSARY; 

RESPONSE  INSTRUCTIONS: 

F/1:  No  configuration  management  of  the  software  test  process  and  its  documentation  has  been 
planned. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SPM(TS)  -  020 

QUESTION:  The  transition  of  the  software  test  strategy  from  the  development  contractor  to  the 
operational  support  activity  has  been  adequately  addressed  in  the  software  test  documentation  and 
the  procurement  software  test  plans. 

ACTIVITY(IES);  All 

EXPLANATION:  All  transition  of  as  much  of  the  software  test  strategy  as  possible  must  he  planned 
from  the  contract  requirements  through  the  program  management  responsibility  transfer.  From  the 
procurement  PMP,  TEMP,  CRLCMP,  RFP,  and  CDRLs  through  development  contractor  and 
operational  support  activity  software  test  planning,  the  transition  must  he  integrated  as  a  critical 
part  of  the  software  deliverable  for  supportability  purposes. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

F/1:  No  transition  of  the  development  contractor  software  test  plan,  test  cases,  test  approach,  or 
test  tools  is  documented. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4,3, 2,1  =  COMPLETELY  DISAGREE) 
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SOFTWARE  PROJECT  MANAGEMENT  PROJECT  INTERFACES 


The  questions  SPM(PI)-001  through  SPM(PI)-016  address  adequacy  of  software  project 
management  external  interfaces  among  the  procurement,  development  contractor,  operational 
support,  and  interservice  organization  elements  as  is  appropriate.  These  project  management 
interfaces  are  established  so  that  project  information  can  be  more  efficiently  communicated.  These 
interfaces  include  the  physical  relationship  among  the  organization  elements  and  the  high-level 
logical  relationship  of  the  organization  elements  to  the  project's  functional  requirements. 

There  are  generally  two  types  of  interfaces  that  are  important  to  evaluate  for  any  project: 
internal  organization  interfaces,  and  organization  interfaces  across  external  boundaries.  The  project 
interface  evaluation  primarily  focuses  on  how  well  the  external  interfaces  and  basic  project  organiza¬ 
tion  to  support  those  interfaces  for  each  major  organization  component  facilities  production  of  a  high 
quality  software  product. 

Characteristics  that  should  be  evaluated  are  the  number  of  external  interfaces,  size  of  interface 
organizations,  various  working  groups'  interfaces,  and  application  of  directives  and  regulations  to 
control  the  coordination  among  interfacing  organizations.  Typical  interface  working  groups  include 
the  computer  resources  working  group  (CRWG),  the  test  planning  working  group  (TPWG),  and  the 
interface  control  working  group  (ICWG). 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PI)  -  001 

QUESTION:  The  system  program  office  external  interfaces  are  adequate. 

ACTIVITYdES):  Procurement 

EXPLANATION:  The  program  office  interfaces  with  the  using  command,  the  operational  support 
activity,  the  training  command,  the  test  and  evaluation  agencies  for  DT&E  and  OT&E,  special 
government  contract  service  agencies  such  as  the  MITRE  Corporation,  and  the  development 
contractor.  These  external  interfaces  generally  are  concerned  with  Air  Force  policy,  adherence  to 
regulations  and  directives,  interservice  implications  of  the  system  under  development,  general 
computer  resource  management  issues,  and  resource  (personnel,  systems,  time,  funds)  planning  and 
use.  The  program  manager  is  the  primary  head  of  the  program  office  and  must  ensure  that  the 
program  office  works  with  the  various  commands  and  agencies  to  establish  the  means  to  implement 
the  system  acquisition  dictated  in  the  PMD  and  described  in  the  PMP.  These  interfaces  help  to 
define  the  technology  constraints  on  the  system  and  its  software,  including  what  advanced  computer 
technology  will  be  required  to  be  applied.  The  initial  emphasis  on  software  supportability  would  be 
in  the  PMP. 

GLOSSARY: 

Program  Office.  An  Air  Force  procuring  activity,  headed  by  a  program  manager,  and 
established  within  a  product  division  (e.g..  Aeronautical  Systems  Division)  early  in  the 
demonstration  and  validation  phase  for  the  purpose  of  fulfilling  the  program  management 
responsibilities  described  in  the  system  PMD, 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PI)  -  002 


QUESTION:  The  implementing  external  interfaces  are  adequate. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  program  office  works  to  make  sure  the  PMP  addresses  proper  development 
issues  and  with  the  operational  support  activity  and  using  command  to  ensure  that  the  operational 
and  support  issues  are  properly  addressed.  If  there  is  a  development  IV&V  function,  then  the 
implementing  command  will  interface  with  the  IV&V  contractor/agency  to  ensure  proper  coordina¬ 
tion  among  the  program  office  and  the  development  contractor  activity.  The  implementing  command 
is  the  primary  contract  monitor  and  technical  reviewer  of  the  development  contractor  activity  tasks. 
The  implementing  command  project  program  manager  must  help  to  define  the  technology 
constraints  on  the  system  and  its  software,  including  what  advanced  computer  technology  will  be 
required  to  be  applied.  The  implementing  command  also  participates  in  working  groups  such  as  the 
CRWG,  ICWG,  and  TPWG.  The  initial  emphasis  on  software  supportability  would  be  in  the  PMP, 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 


132 


AFOTECPAM  99-102  Volume  2  Attachment  1  1  August  1994 


QUESTION  DATA  SHEET 


Question  Number  SPM(PI)  -  003 


QUESTION:  The  using  command  external  interfaces  are  adequate. 

ACTIVITY(IES):  Operational  Support 

EXPLANATION:  The  using  command  has  the  responsibility  of  operational  deployment  of  a  system, 
subsystem,  or  item  of  equipment.  The  using  command  has  external  interfaces  with  the  operational 
support  command,  program  office,  and  various  working  groups.  During  the  acquisition  of  the  system 
to  be  operated,  the  using  command's  primary  interface  role  is  as  monitor  to  assist  in  deriving  the 
system  requirements  necessary  to  make  the  system  operationally  effective.  During  the 
postdeployment  software  support,  the  using  command  interfaces  with  the  operational  support 
activity  and  various  working  groups  concerning  software  block  releases. 

GLOSSARY: 

Using  Command:  The  command  (or  commands  and  contractor  support)  responsible  for  the 
operational  emplo5m[ient  of  the  acquired  system. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PI)  -  004 

QUESTION:  The  operational  support  activity  external  interfaces  are  adequate. 

ACTIVITY(IES):  Operational  Support 

EXPLANATION:  The  operational  support  activity  external  interfaces  include  the  program  office, 
using  command,  working  groups  (CRWG,  ICWG,  TPWG)  as  appropriate.  The  primary  functions  of 
the  interfaces  are  to  ensure  that  the  system  as  delivered  is  supportable  and  the  appropriate  support 
environment  is  acquired  for  use  during  postdeployment  support. 

GLOSSARY: 

Operational  Support  Activity.  The  command/organization  responsible  for  the  configuration 
management,  logistics  support,  and  other  kinds  of  direct  support  required  by  a  system  during 
operational  use. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6,5, 4, 3,2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PI)  -  005 

QUESTION:  The  training  command  external  interfaces  are  adequate. 

ACTIVITY(IES):  Procurement  and  Operational  Support 

EXPLANATION:  The  training  command  reviews  system  documents  and  initiates  training  support 
planning  and  evaluation,  as  appropriate,  and  provides  and  administers  training  programs  to  support 
systems.  The  interfaces  for  such  training  could  include  implementing,  using,  operational  support 
activity,  and  the  DT&E  and  OT&E  agencies. 

GLOSSARY: 

Training  Command.  The  command  (e.g.,  HQ-AETC)  responsible  for  providing  planning,  evalua¬ 
tion,  conduct,  and  administration  of  training  programs  and  training  requirements. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 


AFOTECPAM  99-102  Volume  2  Attachment!  1  August  1994 


135 


QUESTION  DATA  SHEET 


Question  Number  SPM(PI)  -  006 

QUESTION:  The  development  contractor  external  interfaces  are  adequate. 

ACTrVlTY(IES):  Development  Contractor 

EXPLANATION:  The  development  contractor  primary  external  interfaces  are  with  the  program 
office,  implementing  command,  test  and  evaluation  agencies  (DT&E,  OT&E,  IV&V),  operational 
support  activity,  using  command,  and  working  groups  as  is  appropriate.  The  major  function  of  the 
interfaces  are  communication  of  development  requirements  and  status,  conduct  of  integration  and 
operational  tests,  and  transfer  of  the  developed  system  to  the  operational  support  activity. 

GLOSSARY: 

Development  Contractor.  The  prime  contractor  and  any  subcontractors  responsible  for  the  full- 
scale  development  of  the  software  system. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 


136 


AFOTECPAM  99-102  Volume  2  Attachment  1  1  August  1994 


QUESTION  DATA  SHEET 


Question  Number  SPM(PI)  -  007 

QUESTION:  The  development  test  and  evaluation  (DT&E)  organization  external  interfaces  are 
adequate. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  implementing  command  is  responsible  for  DT&E  management.  The 
development  contractor  and  the  implementing  command  jointly  conduct  the  early  part  of  DT&E. 
The  specifics  of  the  DT&E  management  is  documented  in  the  TEMP.  DT&E  data  must  be  provided 
to  the  OT&E  agency.  Some  of  the  functions  of  the  interfaces  are  to  communicate  evaluation  of 
contract  specifications,  system/software  deficiencies,  interoperability  capability,  and  estimates  of  the 
system's  operational  reliability,  supportability,  availability,  and  safety.  See  AFR  80-14  for  more 
information. 

GLOSSARY: 

DT&E  Organization,  The  organization  elements  (primarily  implementing  command  and 
development  contractor)  responsible  for  demonstrating  that  the  system  (including  hardware  and 
software)  design  and  development  is  complete,  that  design  risks  have  been  minimized,  and  that  the 
system  will  perform  as  required  and  specified. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PI)  -  008 

QUESTION:  The  operational  test  and  evaluation  (OT&E)  organization  external  interfaces  are 
adequate. 

ACTIVITY(IES);  Procurement 

EXPLANATION:  The  OT&E  is  managed  and  conducted  by  AFOTEC.  Interfaces  with  nearly  all 
program  organization  elements  are  possible  depending  on  the  system  being  acquired.  Some  of  the 
functions  of  these  interfaces  are  to  communicate  an  estimate  of  the  system's  operational  effectiveness 
and  suitability,  identify  operational  deficiencies,  recommend  changes,  and  identify  system/software 
characteristics  and  deficiencies  that  can  significantly  impact  support  costs.  See  API  99-102,  and 
AFOTECI  99-101  for  more  information. 

GLOSSARY: 

OT&E  Organization.  The  organization  elements  (primarily  AFOTEC  or  other  service  opera¬ 
tional  test  agencies)  responsible  for  the  estimation  of  a  system's  operational  effectiveness  and 
suitability,  identification  of  any  operational  and  support  deficiencies,  and  identification  of  any  need 
for  modifications. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PI)  -  009 

QUESTION;  The  computer  resources  working  group  (CRWG)  external  interfaces  are  adequate. 
ACTIVITY(IES):  Procurement 

EXPLANATION:  For  each  system  utilizing  computer  resources,  a  CRWG  must  be  established 
immediately  after  Milestone  I  to  aid  in  the  management  of  the  system's  computer  resources  and  in 
the  development  of  the  CRLCMP.  The  purpose  of  the  CRWG  is  to  assist  the  program  manager  in 
initiating  activities  that  are  prerequisites  to  development  and  support  of  computer  resources.  The 
CRWG  should  also  assist  in  ensuring  that  computer  resources  comply  with  established  policy, 
procedures,  plans,  and  standards.  The  CRWG  should  include  representatives  of  the  procurement 
activity,  operational  support  activity,  and  also  other  organizations  that  have  been  assigned 
responsibilities  for  software  development,  testing,  training,  and  support. 

GLOSSARY: 

CRWG.  Computer  resources  working  group  is  chaired  by  the  program  office  and  consists  of 
representatives  of  the  participating  commands  and  test  organizations.  Primary  function  is  to 
produce  the  CRLCMP. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PI)  -  010 

QUESTION:  The  test  planning  working  group  (TPWG)  external  interfaces  are  adequate. 
ACTIVITYdES):  All 

EXPLANATION:  The  TPWG  assists  the  program  manager  on  test  matters.  This  assistance 
includes  defining  responsibilities  and  relationships  among  test  program  participants;  establishing 
test  objectives;  preparation  of  the  TEMP;  identifying  test  resources;  preparing  test  portions  of  RFPs, 
SOWs,  and  related  contractual  documents;  and  acting  as  a  forum  for  surfacing  and  resolving  test 
related  issues. 

GLOSSARY: 

TPWG.  The  test  planning  working  group  is  chaired  by  the  implementing  command  with 
representatives  from  the  using  and  supporting  commands,  the  test  organizations  (DT&E,  OT&E), 
and  where  appropriate,  the  development  contractor  and  subcontractors. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3,2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PI)  -  Oil 

QUESTION:  The  interface  control  working  group  (ICWG)  external  interfaces  are  adequate. 

ACTIVITY(IES):  All 

EXPLANATION:  The  procurement  activity  should  establish  an  ICWG  to  address  system/subsystem 
interface  requirements,  including  those  that  affect  computer  resources.  Early  in  the  system 
acquisition  process,  the  ICWG  supports  the  procurement  activity  in  defining  current  and  proposed 
computer  software  and  hardware  interfaces.  The  interface  description  should  include  quantitative 
data  needed  to  accurately  define  interfaces.  Interoperability  requirements  should  be  included  in  the 
interface  definition.  The  procurement,  development  contractor,  and  operational  support  activities 
should  provide  computer  resource  inputs  to  the  ICWG  to  ensure  that  system  interfaces  adequately 
reflect  software  and  hardware  characteristics. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


response  SCORE" 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PI)  -  012 

QUESTION:  The  independent  verification  and  validation  (IV&V)  agency  external  interfaces  are 
adequate. 

ACTIVITY(IES):  All 

EXPLANATION:  The  procurement  activity  should  determine  IV&V  requirements.  The  CRWG  will 
assess  the  need  for  IV&V  and  provide  recommendations  to  the  procurement  activity.  The 
determination  should  consider  the  criticality  of  the  system  mission,  the  risk  associated  with  the 
development,  and  the  level  of  software  complexity.  The  procurement  activity  should  define  the  scope 
and  timing  of  the  IV&V  and  develop  a  plan  for  the  IV&V  effort.  The  IV&V  decisions  should  be 
documented  in  the  CRLCMP.  If  IV&V  is  to  be  included  in  the  software  development,  it  must  be 
initiated  not  later  than  the  award  of  the  development  contract.  Procurement  activities  for  the  IV&V 
effort  should  be  completed  in  advance  of  the  software  specification  review  to  allow  verification  of  the 
software  requirements  before  the  review  is  conducted.  The  procurement  activity  shall  control  the 
interface  between  the  IV&V  agency  and  the  development  contractor  and  provide  the  IV&V  agency 
with  copies  of  the  appropriate  development  specifications,  design  documents,  listings,  and 
discrepancies  found  by  the  IV&V  agency. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE; 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PI)  -  013 

QUESTION:  The  software  configuration  management  interfaces  among  all  activities'  management 
components  for  the  subject  system  are  adequate. 

ACTIVITY(IES):  All 

EXPLANATION:  Each  of  the  activities  has  a  responsibility  for  configuration  management  that 
requires  interface  communication.  Assignment  of  identification  numbers  is  a  procurement  and 
operational  support  responsibility  dependent  on  the  development  contractor  formal  request  for  such 
numbers.  The  procurement  activity  maintains  the  formal  baseline  (functional,  allocated,  product) 
while  the  development  contractor  maintains  developmental  baseline  up  to  delivery  of  the  product 
baseline.  Changes,  refinements,  and  so  forth  must  be  communicated.  The  procurement  and 
development  contractor  must  coordinate  transfer  of  configuration  management  responsibility  as 
defined  in  the  CRLCMP  with  the  operational  support  activity, 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PI)  -  014 

QUESTION:  The  software  quality  assurance  management  interfaces  among  all  activities'  manage¬ 
ment  components  for  the  subject  system  are  adequate. 

ACTIVITY(IES):  All 

EXPLANATION:  The  overall  software  quality  program  for  the  computer  software  development 
cycle  should  be  defined  by  the  procurement  activity  and  operational  support  activity  through 
contractual  requirements.  Responsibility  for  assessing  computer  software  products  and  related 
procedures  may  be  assigned  to  more  than  one  organization.  It  may  be  appropriate  for  an 
independent  organization,  such  as  an  IV&V  organization,  that  is  subject  to  neither  financial  nor 
managerial  control  by  the  development  contractor,  to  perform  certain  of  these  assessments. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PI)  -  015 

QUESTION:  The  contract  management  interfaces  among  all  activities'  management  components 
for  the  subject  system  are  adequate. 

ACTIVITY(IES):  All 

EXPLANATION:  Contractual  provisions  should  reflect  the  government’s  requirements  for  rights  to 
the  computer  software  and  associated  documentation.  Development  contractor  limited-rights 
software  to  be  used  in  the  performance  of  the  contract  or  to  be  delivered  under  the  contract  should  be 
identified.  Predetermination  agreements  should  be  included  in  the  contract  to  enable  the 
government  to  subsequently  acquire  additional  required  rights.  Because  computer  resources 
(including  computer  software)  may  be  developed  under  a  subcontract  to  a  prime  contractor,  the 
procurement  activity  should  ensure  that  all  appropriate  contractual  requirements  levied  on  the 
prime  development  contractor  are  passed  to  the  subcontractor.  The  procurement  activity  should 
ensure  that  the  contract  makes  the  subcontractor  responsible  for  the  integrity  of  subcontracted 
products  and  makes  the  prime  development  contractor  responsible  for  delivery  of  an  acceptable 
product  under  the  contract. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPM(PI)  -  016 

QUESTION:  The  inter  service  external  interfaces  with  all  activities*  management  components  are 
adequate. 

ACTIVITY(IES):  All 

EXPLANATION:  Before  each  system  milestone,  interservicing  potential  should  be  reviewed  and 
the  management  and  life  cycle  cost  implications  of  major  software  support  options  should  be 
analyzed.  This  analysis  should  also  consider  impact  on  operational  needs,  configuration 
management,  and  system  integration.  For  interservice  systems,  the  CRWG  and  ICWG  shall  be 
interservice  groups  that  include  representatives  from  the  cognizant  participating  organization 
elements  (commands  agencies,  etc.).  The  interservice  working  groups  should  ensure  that  analysis  is 
performed  to  determine  the  optimum  support  approach,  this  analysis  is  documented,  and 
recommendations  are  made  to  the  procurement  activity  concerning  the  support  approach. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

A/6:  There  are  no  joint  service  requirements  for  the  system  or  its  embedded  software. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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SOFTWARE  CONFIGURATION  MANAGEMENT  IDENTIFICATION 


The  questions  SCM(ID)-001  through  SCM(ID)-021  address  issues  of  software  configuration 
identification  for  the  procurement,  development  contractor,  and  operational  support  activities. 
Configuration  identification  is  established  in  the  form  of  technical  documentation  that  becomes  more 
detailed  as  development  proceeds  and  more  refined  as  the  final  development  products  are  evolved 
during  postdeployment  support.  Three  stages  of  configuration  identification  are  generally  employed 
during  the  software  system's  life  cycle: 

(1)  Functional  Configuration  Identification 

(2)  Alocated  Configuration  Identification 

(3)  Product  Configuration  Identification 

In  addition,  the  development  contractor  activity  has  internal  iterations  of  the  identifications  called 
Development  Identifications  which  are  controlled  by  the  necessary  internal  software  configuration 
management  process. 

These  identifications  correspond  to  the  system  development  baselines: 

(1)  Functional  Baseline 

(2)  Alocated  Baseline 

(3)  Product  Baseline 

and,  the  appropriate  development  contractor  development  baselines.  The  Identifications  become 
Baselines  when  the  procurement  activity  approves  the  Identifications  and  puts  the  configuration 
identification  under  its  contractual  configuration  control  system.  Identification  is  used  for  visibility, 
and  baselines  are  used  for  control. 

The  term  "identification"  also  has  an  important  secondary  meaning  as  a  document  or  set  of 
documents  that  defines  the  configuration  of  an  item.  In  this  sense,  it  represents  one  or  more 
material  things  (documents). 


AFOTECPAM  99-102  Volume  2  Attachment  1  1  August  1994 


147 


QUESTION  DATA  SHEET 


Question  Number  SCM(ID)  -  001 

QUESTION:  The  procurement  policy,  standards,  and  conventions  applied  to  the  identification  of 
software  configuration  items  are  adequate. 

ACTIVITYdES):  Procurement 

EXPLANATION:  Identification  of  computer  software  configured  items  and  procedures  for  assigning 
identification  numbers/names  is  described  in  APR  800-21.  It  is  the  responsibility  of  the  procurement 
activity  to  ensure  that  proper  policy,  standards,  and  conventions  are  required  for  the  naming  of 
configured  items.  Directive  and  guidance  documents  include  APR  800-14.  Contractor  compliance 
documents  that  could  be  required  include  MILrSTD-480A,  MIL-STD-482A,  MIL-STD-483A,  and  MIL- 
STD-490A,  and  DoD-STD  2167A. 

GLOSSARY: 

Software  Configuration  Items.  Software  elements  that  are  designated  for  configuration  control 
by  the  contractual  requirements. 

Identification.  The  official  character/numeric  identifier  of  a  configured  item  and  its  functional 
purpose  and  relationship  with  other  configured  items  for  purposes  of  configuration  management. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5,4, 3,2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(ID)  -  002 

QUESTION:  The  procurement  identification  of  deliverable  software  configuration  items  is 

adequate. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  Frequently,  software  items  are  not  required  to  be  delivered.  The  CDRL  is  the 
basis  for  delivery.  The  CDRLs  should  carefully  identify  all  operational,  test,  and  support  software 
deliverable.  Included  with  this  identification  is  the  form  of  the  deliverable  (e.g.,  document,  source, 
object,  load  module)  and  the  medium  on  which  the  deliverable  will  be  produced.  Include  in  the 
contract  all  software  including  firmware  and  proprietary  items  that  are  required  to  cost  effectively 
use,  operate,  or  modify  the  system  over  its  life  cycle.  If  it  is  not  cost  effective  to  acquire  a  software 
item,  include  an  option  to  acquire  it  later. 

GLOSSARY: 

Deliverable  Software.  Identified  by  CDRLs. 

Operational  Software.  Required  to  operate  the  system. 

Test  Software.  Used  to  analyze  or  test  system  and  component  performance. 

Support  Software.  Used  generally  to  develop  or  maintain  other  software.  Includes  system 
software  such  as  operating  systems  and  compilers. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(ID)  -  003 

QUESTION;  The  procurement  activity  identification  of  the  software  configuration  baselines  is 
adequate. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  software  configuration  baselines  include  the  Functional  Baseline,  Allocated 
Baseline,  and  Product  Baseline.  When  the  procurement  activity  approves  the  development 
Configuration  Identification  for  each  of  these  baselines,  then  the  Identification  becomes  the 
corresponding  Baseline  and  is  put  under  the  procurement  activity  configuration  control.  This 
requires  the  procurement  activity  to  have  adequate  identification  capability  to  maintain  such 
configuration  control. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(ID)  -  004 

QUESTION:  The  system/segment  specification  adequately  identifies  elements  of  the  software 
functional  baseline. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  system/segment  specification  is  a  formally  controlled  software  item  of  a 
system  procurement.  This  specification  defines  the  Functional  Configuration  Identification  of  the 
software.  The  first  version  is  usually  prepared  by  the  procurement  activity  and  becomes  a  living 
document  of  the  system/segment  performance-oriented  requirement.  When  it  is  approved  by  the 
procurement  activity,  it  is  "baselined"  and  comes  under  configuration  control  as  the  Functional 
Baseline.  The  Functional  Baseline  should  be  available  at  Milestone  I,  prior  to  Demonstration  and 
Validation.  It  is  critical  that  this  specification  reflect  as  complete  a  perspective  on  the  functional 
aspects  of  the  system  as  possible.  It  is  also  critical  that  this  specification  mature  as  early  as  possible 
to  minimize  perturbations  on  the  rest  of  the  system  baselines. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE^ 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(ID)  -  005 

QUESTION:  The  performance  requirement  specifications  adequately  identify  elements  of  the 
software  allocated  baseline. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  performance  requirement  specifications  are  the  descriptions  of  how  the 
Functional  Baseline  is  allocated  into  CSCIs  (Computer  Software  Configuration  Items)  and  HWCIs 
(Hardware  Configuration  Items).  These  specifications  include  preliminary  requirement  documents 
and  become  living  documents.  When  they  are  approved  by  the  procurement  activity,  they  are 
'‘baselined"  as  the  Allocated  Baseline.  The  Allocated  Baseline  should  be  available  at  Milestone  II, 
prior  to  engineering  and  manufacture  development.  It  is  critical  that  these  specifications  reflect  as 
complete  a  perspective  on  the  detailed  function  allocation,  test,  and  interface  aspects  of  the  system  as 
possible.  It  is  also  critical  that  these  specifications  mature  as  early  as  possible  to  minimize 
engineering  and  manufacture  development  perturbations. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(ID)  -  006 

QUESTION:  The  implementation  specifications  adequately  identify  elements  of  the  software 
product  baseline. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  implementation  specifications  are  the  description  of  how  the  Allocated 
Baseline  has  been  implemented  as  CSCIs  (Computer  Software  Configuration  Items)  and  HWCIs 
(Hardware  Configuration  Items).  These  implementation  specifications  include  the  "as  built"  detailed 
design  documents  and  become  living  documents.  When  they  are  approved  by  the  procurement 
activity,  they  are  "baselined"  and  become  the  Product  Baseline.  It  is  critical  that  these  specifications 
reflect  as  complete  a  perspective  on  the  detailed  design  and  coded  aspects  of  the  system  as  possible. 
It  is  also  critical  that  these  specifications  mature  as  early  as  possible  to  facilitate  the  transfer,  opera- 
tion,  and  support  of  the  software. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 

Question  Number  SCM(ID)  -  007 

QUESTION:  The  identifier  characteristics  for  software  configuration  item  names  are  adequate. 
ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  identifier  characteristics  include  uniqueness,  retrievability,  traceability, 
pronouncibility,  variability,  functional  significance,  and  compactness. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6,5,4, 3,2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(ID)  -  008 

QUESTION:  The  development  contractor  internal  identifier  naming  standards/conventions  satisfy 
contractual  regulations. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  development  contractor  is  (should  be)  required  to  follow  Air  Force  regulations 
on  Computer  Program  Identification  Number  (CPIN)  assignments.  Air  Force  directive  guidance  is 
found  in  APR  800-14,  along  with  other  documents.  The  development  contractor  compliance 
documents  are  DoD-STD-480A,  MIL-STD-482A,  MIL-STD-483A,  and  MILi-STD-490A. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(ID)  -  009 

QUESTION:  Development  contractor  identification  standards  and  conventions  can  be  transitioned 
to  operational  support  standards  and  conventions. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  In  order  for  computer  resources  to  be  smoothly  transitioned  from  the 

development  contractor  to  the  operational  support  activity,  the  configuration  identification  standards 
and  conventions  must  be  compatible.  As  more  automated  tools  are  used,  this  requirement  for 
compatibility  will  be  even  stronger.  Evidence  of  the  standards  and  transition  strategy  should  be  in 
the  CRLCMP  as  well  as  the  development  contractor  software  configuration  management  plan. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

F/1:  No  standards  exist  for  the  development  contractor  activity  and/or  the  operational  support 

activity. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(ID)  -  010 

QUESTION:  Development  contractor  deliverable  configuration  items  are  named  to  adequately 
identify  multiple  versions  and  variations. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  minimum  requirement  is  for  the  name  to  provide  for  discrimination  of 
versions.  If  software  must  be  configured  specifically  for  test  purposes,  multiple  sites,  and  so  forth,  it 
will  be  necessary  for  the  name  to  distinguish  such  variations  of  each  version. 

GLOSSARY: 

Version.  Baseline  release  of  a  configuration  controlled  item. 

Variation.  One  of  at  least  two  physical  configurations  of  the  same  version  of  a  configuration 
controlled  item.  Variations  of  a  version  exist  to  support  multiple  service  requirements  as  well  as 
mission  specific  configurations  (test,  operational  mission  scenarios,  alternate  embedded  computer 
systems,  etc.). 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6,5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(ID)  -  Oil 

QUESTION:  Development  contractor  identification  procedures  are  structured  to  permit  easy 
addition,  deletion,  or  modification  of  configured  items  at  any  hierarchical  level, 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  identification  procedures  should  be  specified  in  the  Software  Configuration 
Management  Plan  or  perhaps  a  set  of  procedures  to  implement  portions  of  the  Plan.  Hierarchical 
levels  are  CSCI,  component,  module,  and  routine.  Such  procedures  are  essential  for  adequate 
software  supportability. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

F/1:  No  such  procedures  are  documented  or  are  totally  inadequate. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(ID)  -  012 

QUESTION:  Development  contractor  identification  procedures  for  addition,  deletion,  and 

modification  of  configured  items  are  being  followed. 

ACTrVlTY(IES):  Development  Contractor 

EXPLANATION:  It  should  be  possible  to  determine  whether  identification  procedures  are  being 
followed  through  the  standard  management  reporting  requirements. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

F/1:  No  such  procedures  exist  or  are  being  totally  ignored. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(ID)  -  013 

QUESTION:  The  physical  medium  of  configured  items  is  adequately  described  by  the  development 
contractor  software  component/item  identification  scheme. 

ACTIVITYdES):  Development  Contractor 

EXPLANATION:  The  physical  medium  (e.g.,  tape,  disk,  memory  components)  of  configured  items  is 
required  to  meet  government  standards.  Part  of  those  standards  address  identification 
names/numbers.  It  should  be  possible  to  trace  the  medium  of  a  configured  item  from  its  descriptive 
label/name  and  any  distinguishing  aspects  of  the  medium  (e.g.,  working/master  tape,  sequential 
volume  number  for  multivolume  storage  items). 

GLOSSARY: 

Medium.  Disk,  tape,  card  deck,  firmware,  read-only  memory,  etc. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4,3,2, 1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(ID)  -  014 

QUESTION:  The  development  contractor  software  identifiers  adequately  distinguish  among 
different  states  (e.g.,  course,  object,  load,  core  images,  listings)  of  the  software. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  identifier  should  distinguish  which  state  the  software  item  is.  For  example,  a 
distinguishing  suffix  might  be  attached  to  the  software  item  identifier,  such  as:  ".prg,"  ".txt,"  ".cmd," 
".dat." 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(ID)  -  015 

QUESTION:  The  development  contractor  software  change  control  form  is  adequate. 
ACTIVITY(IES):  Development  Contractor. 

EXPLANATION:  The  change  control  form  identifiers  should  meet  requirements  of  applicable 
government  standards  and  provide  sequential  identification  suitable  for  logging,  filing,  reference, 
and  retrieval. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5,4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(ID)  -  016 

QUESTION:  Subcontractor  configuration  item  identification  practices  are  monitored  by  the 
development  contractor. 

ACTrVlTY(IES):  Development  Contractor 

EXPLANATION:  If  there  is  a  subcontractor,  it  will  be  necessary  that  the  development  contractor 
require  configuration  identification  practices  similar  to  those  required  by  the  procurement  activity. 
If  this  is  not  done,  then  the  development  contractor  will  be  required  to  retrofit  the  identification 
scheme  of  the  subcontractor.  The  identification  practices  of  the  subcontractor  must  be  carefully 
monitored  to  ensure  compatibility. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

A/6:  No  subcontractors  are  involved  with  producing  software  configuration  items  for  the 
development  contractor. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(ID)  -  017 

QUESTION:  The  documentation  that  collectively  identifies  the  content  of  a  configuration  item  is 
adequate. 

ACTIVITY(IES):  Development  Contractor. 

EXPLANATION:  The  documentation  might  include  a  version  description  document  or  a  software 
configuration  index.  The  version  description  document  usually  identifies  changes  to  a  baseline 
product  components  (e.g.,  in  a  hierarchical  chart)  showing  component. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(ID)  -  018 

QUESTION:  Software  configured  items  that  implement  safety  provisions  are  adequately  identified. 
ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  Configured  items  that  implement  safety  provisions  are  frequently  controlled  by 
software.  This  software  must  be  adequately  identified  as  affecting  safety.  Safety  provisions  are 
closely  related  to  the  reliability  of  mission  critical  components,  safety  of  mission  personnel,  nuclear 
effects,  and  so  forth. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

A/6:  There  is  no  requirement  for  safety  provisions  to  be  implemented  or  controlled  by  software. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(ID)  -  019 

QUESTION:  Software  configured  items  that  implement  computer/communication  security 

provisions  are  adequately  identified. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  Software  that  implements  computer/communication  security  is  particularly 
important.  Any  such  software  items  must  be  adequately  identified  as  part  of  the  trusted  computer 
base.  If  the  configured  software  items  are  themselves  classified,  then  appropriate  security  labels 
must  be  attached  according  to  Air  Force  labeling  requirements. 

GLOSSARY: 

Security  Provisions.  The  totality  of  threats,  vulnerabilities,  and  protection  mechanisms  involved 
with  determining  whether  computer/communications  assets  can  be  compromised  through  data, 
process,  or  abuse  violations.  Security  provisions  exist  across  the  administrative  system,  and  facility 
categories. 

RESPONSE  INSTRUCTIONS: 

A/6:  There  is  no  requirement  for  security  provisions  to  be  implemented  in  software. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(ID)  -  020 

QUESTION:  The  identification  requirements  for  postdeployment  support  are  adequately  addressed 
in  the  CRLCMP, 

ACTIVITY(IES):  Operational  Support. 

EXPLANATION:  The  Computer  Resources  Life  Cycle  Management  Plan  or  CRLCMP  is  the 
planning  document  for  operational  support  configuration  management.  The  CRLCMP  is  intended  to 
be  a  living  document,  evolving  to  provide  a  current  view  of  the  support  evolution  of  the  computer 
resources  including  configurations  management  features.  The  CRLCMP  (first  version)  is  required 
early  in  the  life  cycle,  usually  in  concept  exploration  phase.  Key  to  adequacy  is  the  compatibility  of 
the  operational  support  configuration  identification  and  the  development  contractor  configuration 
identification, 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(ID)  -  021 

QUESTION:  The  automated  support  tools  for  postdeployment  support  of  configuration 

identification  are  adequately  addressed  in  the  CRLCMP. 

ACTIVTTY(IES):  Operational  Support 

EXPLANATION:  The  Computer  Resources  Life  Cycle  Management  Plan  or  CRLCMP  is  the  key 
planning  document  for  operational  support  configuration  management.  The  CRLCMP  is  intended  to 
be  a  living  document,  evolving  to  provide  a  current  view  of  the  configuration  management  features 
along  with  the  evolution  of  the  system.  The  use  of  automated  support  tools  during  development  and 
transition  of  those  tools  to  use  during  postdeployment  support  is  an  important  consideration  for  the 
overall  enhancement  of  software  supportability.  The  lack  of  such  tools  to  manage  the  configuration 
identification  index  of  the  various  baselines  should  be  considered  a  serious  deficiency. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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SOFTWARE  CONFIGURATION  MANAGEMENT  CONTROL 


The  questions  SCM(CC)-001  through  SCM(CC)-023  address  issues  of  software  configuration 
control  for  the  procurement,  development  contractor,  and  operational  support  activities. 
Configuration  control  is  the  major  process  of  configuration  management.  It  is  the  process  by  which 
change  decisions  are  made  (by  the  Configuration  Control  Board  structure),  administered  (by  the 
Configuration  Management  Office  of  the  Program  Office—or  equivalent),  and  implemented  (by 
change  control  personnel  appropriate  to  the  life  cycle  state  of  the  software). 

The  decision-making  part  of  configuration  control  determines  whether  proposed  changes  to  a 
controlled  document  or  software  item  will  be  beneficial  to  the  government  in  terms  of  operational 
effectiveness,  support  needs,  cost,  and/or  schedule.  The  change  administration  and  implementation 
parts  ensure  that  all  approved  changes  to  a  configuration  are  properly  incorporated  in  the  affected 
documents  and  software  code  and  that  no  other  changes  find  their  way  in. 

Configuration  control  focuses  on  the  approved  baselines: 

(1)  Functional  Baseline 

(2)  Allocated  Baseline 

(3)  Product  Baseline 

and,  the  appropriate  development  contractor  development  baselines. 

Software  items  and  documents  that  are  not  baselined  are  not  subject  to  baseline  configuration 
control,  but  may  be  placed  under  internal  (contractor  or  support)  configuration  control  during  the 
software  life  cycle.  Baseline  configuration  control  relies  on  interaction  among  the  procurement, 
development  contractor,  and  operational  support  activities.  The  adequacy  of  the  development 
contractor  internal  configuration  control  is  important  since  the  plans,  techniques,  and  tools  would  be 
beneficial  for  transfer  to  the  operational  support  activity  for  use  during  the  postdeployment  life  cycle 
phase.  From  the  operational  support  viewpoint,  it  is  not  sufficient  that  the  development  contractor 
activity  can  control  the  baselines  sufficiently  to  deliver  a  configured  product.  For  smooth  transition, 
it  is  necessary  that  the  configuration  control  process  can  be  transitioned  to  or  is  compatible  with  the 
support  activity  configuration  control  process. 

Interface  control  is  also  a  very  important  aspect  of  configuration  control,  especially  with  systems 
that  have  multiservice  operational  requirements  and  systems  that  require  more  than  one  element  for 
development  or  support.  Separate  control  boards  and  review  boards  and  integrated  working  groups 
are  required  to  manage  the  complicated  development  and  support  requirements. 
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QUESTION  DATA  SHEET 


Question  Number  SCM(CC)  -  001 

QUESTION;  The  procurement  policy,  standards,  and  conventions  applied  to  the  control  of  software 
configuration  items  are  adequate. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  Control  of  computer  software  configured  items  and  procedures  for  changing 
configured  items  are  described  in  directives,  regulations,  and  guidelines.  It  is  the  responsibility  of 
the  procurement  activity  to  ensure  that  proper  policy,  standards,  and  conventions  are  required  for 
the  control  of  configured  items.  Contractor  compliance  documents  that  could  be  required  include 
MIL-STD-480A,  MIL-STD-482,  MIL-STD-483A,  MILSTD-490A,  and  DoD-STD-2167A. 

GLOSSARY: 

Software  Configuration  Items.  Software  elements  that  are  designated  for  configuration  control 
by  the  contractual  requirements. 

Control.  The  process  of  systematic  oversight  of  changes  to  a  configured  item  and  its  functional 
relationship  with  other  configured  items  for  purposes  of  configuration  management. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(CC)  -  002 

QUESTION:  The  procurement  activity  has  implemented  adequate  software  configuration 

management,  based  on  regulations,  to  control  the  functional  and  physical  characteristics  of  all 
CSCIs. 

ACTIVITY(IES):  Procurement 

EXPLANATION;  The  procurement  program  manager  is  responsible  for  implementing  a  configura¬ 
tion  management  program  that  will  identify,  document,  and  control  the  functional  and  physical 
characteristics  of  all  CSCIs  under  development.  Primary  planning  document  is  the  Program 
Management  Plan  (PMP).  Other  activities  include  coordinating  requirements  with  using  and 
supporting  agencies,  reviewing  contractor  plans,  auditing  contractor  implementation  of  plans, 
ensuring  configuration  identifications  for  all  CSCIs  are  properly  documented,  controlling  engineering 
changes  to  baselines,  providing  interface  control  for  distribution  of  changes,  and  preparing  for 
transfer  to  the  operational  support  activity. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(CC)  -  003 

QUESTION;  The  procurement  configuration  management  planning  documents  contain  sufficient 
guidance  for  configuration  control, 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  major  planning  documents  for  procurement  are  the  Program  Management 
Plan  (PMP),  the  Request  for  Proposal  (RFP)/Statement  of  Work  (SOW),  the  Contract  Data 
Requirements  List  (CDRL),  and  the  Computer  Resources  Life  Cycle  Management  Plan  (CRLCMP). 
APR  800-14  calls  for  the  inclusion  of  configuration  management  concepts  in  the  PMP  including 
configuration  control  (specification  and  interfaces).  The  RFP/SOW  defines  the  exact  scope  of  the 
development  contractor's  configuration  control  responsibilities.  The  CDRL  identifies  all  deliverable 
data  items  including  CSCIs  that  the  development  contractor  must  deliver  and  control.  The  CRLCMP 
must  also  include  assignment  of  configuration  control  responsibilities  during  postdeployment 
software  support  with  the  detailed  procedures. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 


j 
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QUESTION  DATA  SHEET 


Question  Number  SCM(CC)  -  004 

QUESTION:  The  development  contractor  configuration  management  activities  are  adequately 
monitored  by  the  procurement  activity. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  procurement  activity  can  monitor  development  contractor  configuration 
management  activities  through  contractor  documents  and  reports,  other  program  office  areas  (e.g., 
quality  assurance),  configuration  audits,  and  evaluation  checklists  (e.g.,  PCA  and  PCA  preparation 
checklists  in  MIL-STD-1521B,  ECP  preparation  checklists  in  DoD-STD-480A  and  modified  by  MIL- 
STD-483A). 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3,2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(CC)  -  005 

QUESTION:  The  procurement  configuration  control  procedures  for  the  Class  I  and  Class  II  changes 
(or  equivalent  categories)  are  adequate. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  Class  I  changes  involve  primarily  any  major  changes  to  the  Configuration 
Baselines,  contractual  provisions,  support  compatibility,  and  so  forth.  Class  II  changes  are  for  minor 
deficiencies  and  do  not  generally  require  procurement  approval,  but  there  should  still  exist  a 
mechanism  for  procurement  review  since  many  times  Class  II  changes  could  cause  side  effects  that 
might  result  in  the  change  being  reclassified  as  a  Class  I  change.  A  Class  II  change  is  justified  if  it 
benefits  the  development  contractor  and  is  not  detrimental  to  the  government  (procurement  and 
operational  support).  Guidance  for  Class  I  and  Class  II  changes  is  found  in  DoD-STD-480A,  MIL- 
STD-483 A,  and  other  government  internal  and  compliance  documents. 

GLOSSARY: 

Class  I  Change.  Engineering  change  that  affects  a  Baseline  Identification,  performance  outside 
stated  tolerance,  external  interface  characteristics,  budget/resource  requirements,  or  other  factors  of 
major  significance  to  the  operational  effectiveness  or  suitability  of  the  software  product. 

Class  II  Change.  Engineering  change  not  classified  as  Class  I.  Included  minor  changes  such  as 
typographical  errors  in  documents,  addition  of  comments  to  source  code,  changes  to  adaptation  data 
such  as  data  base  parameters,  and  single  recompilations. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(CC)  -  006 

QUESTION:  The  use  of  deviations  and  waivers  by  the  development  contractor  that  could  affect  the 
supportability  of  the  software  has  been  adequately  controlled  by  the  procurement  activity. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  use  of  deviations  and  waivers  must  be  carefully  monitored  for  the  possible 
adverse  affect  on  software's  supportability  even  though  the  operational  effectiveness  may  not  seem  to 
be  directly  affected.  As  an  example,  if  the  use  of  a  High  Order  Language  is  waived,  then  the 
supportability  of  the  software  has  been  affected. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(CC)  -  007 

QUESTION;  The  procurement  baseline  control  forms  are  adequate. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  procurement  baseline  control  forms  might  include  Engineering  Change 
Proposal  (ECP),  Specification  Change  Notice  SCN),  Request  for  Waiver,  Software  Deficiency  Report, 
and  Software  Change  Report. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5,4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(CC)  -  008 

QUESTION:  The  procurement  configuration  control  board  procedures  are  adequate. 
ACTIVITY(IES):  Procurement 

EXPLANATION:  The  procurement  CCB  procedures  include  conduct  of  meetings,  maintenance  of 
records,  control  of  the  baselines,  integration  of  hardware  and  software  concerns  during  the  change 
process,  formation  of  a  separate  software  configuration  control  board  if  the  complexity  of  the 
software  so  justifies  such  a  board,  and  control  of  interoperability  interface  problems  across  any 
associated  systems.  Change  control  procedures  should  provide  for  careful  evaluation  of  all  ECPs 
according  to  existing  configuration  management  directives.  In  particular,  CSCI  changes  that  have 
an  effect  on  multiple-location  applications,  nuclear  safety,  security,  cost,  schedule,  other  CSCIs,  other 
hardware  or  interfaces,  and  support  resources  must  be  carefully  analyzed  for  overall  benefit  to  the 
government. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5,4,3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(CC)  -  009 

QUESTION:  The  procurement  procedures  for  turnover  and  transfer  of  configuration  control  to  the 
operational  support  activity  have  been  adequately  planned. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  CRLCMP  should  contain  specific  guidance  as  to  the  form  and  format  of 
transition  to  the  operational  support  activity.  Key  to  the  adequacy  of  this  process  is  the  amount  of 
early  planning  and  the  specificity  of  the  details  in  the  CRLCMP  at  the  milestone  (and  interim 
milestone)  decision  points.  Frequently,  the  mere  existence  of  such  documents  does  not  imply  that 
they  are  at  all  adequate. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(CC)  -  010 

QUESTION:  Development  contractor  configuration  control  standards  and  conventions  can  be 
transitioned  to  operational  support  standards  and  conventions. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  CRLCMP  should  contain  specific  guidance  as  to  the  form  and  format  of  the 
transition  to  the  operational  support  activity.  Key  to  the  adequacy  of  this  process  is  the  amount  of 
early  planning  and  the  specificity  of  the  details  in  the  CRLCMP  at  the  milestone  (and  interim 
milestone)  decision  points.  Frequently,  the  mere  existence  of  such  documents  does  not  imply  that 
they  are  at  all  adequate. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(CC)  -  Oil 

QUESTION;  The  development  contractor  configuration  control  board  has  an  adequate  interface 
with  the  procurement  activity  configuration  control  board. 

ACTIVITY(rES):  Development  Contractor 

EXPLANATION:  AFR  800-14  requires  computer  program  configuration  management  to  be 
integrated  into  the  overall  system  configuration  management  and  across  all  cognizant  organization 
elements.  Interfaces  are  very  important,  and  one  of  the  most  important  is  the  communication 
between  the  procurement  and  development  contractor  configuration  control  boards.  The  CCBs  are 
the  official  organizations  empowered  to  act  on  all  proposed  changes.  The  primary  changes  that 
would  require  interfacing  are  Class  I  changes.  MIL-STD-483A  is  the  primary  development 
contractor  compliance  regulation. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(CC)  -  012 

QUESTION;  The  development  contractor  configuration  control  board  procedures  are  adequate  to 
distinguish  between  hardware  and  software  failures. 

ACTIVITYdES):  Development  Contractor 

EXPLANATION:  For  large  systems,  separate  hardware  and  software  boards  may  be  established 
under  a  system  level  board.  Failure  reporting  must  adequately  characterize  failures  so  determina¬ 
tion  of  the  source  of  the  failure  is  possible.  Such  reports  and  solutions  to  failures  can  then  be 
processed  more  adequately  by  the  control  boards. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS; 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(CC)  -  013 

QUESTION;  The  development  contractor  configuration  control  procedures  can  be  transitioned  to  or 
are  compatible  with  the  operational  support  activity  planned  configuration  control  procedures. 

ACTIVITYdES):  Development  Contractor 

EXPLANATION:  The  CRLCMP  describes  the  operational  support  planned  configuration  control 
procedures,  usually  in  accordance  with  APR  57-4.  Contractor  compliance  documents  include  DoD- 
STD-480A  and  MILrSTD-483A.  The  contractor's  configuration  control  procedures  should  be 
documented  in  a  Software  Configuration  Management  Plan. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(CC)  -  014 

QUESTION:  The  development  contractor  automated  support  tools  for  configuration  control  of 
baselines  and  internal  development  identifications  are  adequate. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  automated  support  tools  might  include  a  control  library,  automated 
procedures  to  lock  out  multiple  personnel  from  modifying  a  module  at  the  same  time,  automated 
version  and  variation  identification,  automated  traceability  of  requirements,  design,  code,  test 
elements,  and  so  forth.  The  use  of  automated  tools  is  essential  for  complex  software  systems. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(CC)  -  015 

QUESTION:  The  development  contractor  software  change  control  forms  are  adequate. 
ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  change  control  forms  should  meet  requirements  of  applicable  government 
standards,  and  provide  sequential  identification  suitable  for  logging,  filing,  reference,  and  retrieval. 
Content  should  adequately  address  source  of  change  request,  reason  for  request,  type  (enhancement, 
correction)  of  request,  effect  of  change  on  the  system,  resource  requirements  to  implement  the 
change,  and  administrative  information  such  as  approval  signatures  required  and  expected  (actual) 
change  milestone  dates. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6,5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(CC)  -  016 

QUESTION:  Subcontractor  configuration  item  control  practices  are  monitored  by  the  development 
contractor. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  If  there  is  a  subcontractor,  it  will  be  necessary  that  the  development  contractor 
require  configuration  control  practices  similar  to  those  required  by  the  procurement  activity.  If  this 
is  not  done,  then  the  development  contractor  will  be  required  to  retrofit  the  control  practices  of  the 
subcontractor.  The  control  practices  of  the  subcontractor  must  be  carefully  monitored  to  ensure 
compatibility  and  proper  interfaces  of  the  cognizant  configuration  control  boards. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

A/6:  No  subcontractors  have  responsibility  for  development  of  software  products. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(CC)  -  017 

QUESTION:  Configured  items  that  implement  safety  provisions  are  adequately  controlled. 
ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  Configured  items  that  implement  safety  provisions  are  frequently  controlled  by 
software.  This  software  must  be  adequately  identified  as  affecting  safety.  Safety  provisions  are 
closely  related  to  the  reliability  of  mission  critical  components,  safety  of  mission  personnel,  nuclear 
effects,  and  so  forth. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

A/6:  There  is  no  requirement  for  safety  provisions  to  be  implemented  or  controlled  by  software. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE; 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(CC)  -  018 

QUESTION:  Software  configured  items  that  implement  computer/communications  security 

provisions  are  adequately  controlled. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  Software  that  implements  computer/communication  security  is  particularly 
important.  Any  such  software  items  must  be  adequately  controlled  as  part  of  the  trusted  computer 
base.  If  the  configured  software  items  are  themselves  classified,  then  appropriate  security  labels 
must  be  attached  according  to  Air  Force  labeling  requirements,  and  access  control  of  such  items  must 
be  enforced. 

GLOSSARY: 

Security  Provisions.  The  totality  of  threats,  vulnerabilities,  and  protection  mechanisms  involved 
with  determining  whether  computer/communications  assets  can  be  compromised  through  data, 
process,  or  abuse  violations.  Security  provisions  exist  across  the  administrative,  system,  and  facility 
categories. 

RESPONSE  INSTRUCTIONS: 

A/6:  There  is  no  requirement  for  security  provisions  to  be  implemented  in  software. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SCM(CC)  -  019 

QUESTION:  Distribution  of  configured  item  changes  from  the  operational  support  activity  to  the 
field  is  adequately  controlled. 

ACTIVITY(IES):  Operational  Support 

EXPLANATION:  The  distribution  must  satisfy  applicable  standards  and  regulations  for  technical 
orders  as  well  as  the  mission  critical  issues  of  correctness  of  changes  and  timeliness  of  changes. 
Interfaces  among  operational  support  and  field  support  organization  elements,  including  configura¬ 
tion  boards  and  logistics  supply  for  technical  orders,  are  critical  to  the  success  of  the  distribution 
process. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

Timeliness  of  the  distribution  process  after  engineering  release  is  complete  is  one  of  the  critical 
issues  to  consider.  Although  there  are  no  fixed  standards,  it  seems  reasonable  that  no  more  than  50 
percent  of  the  time  spent  for  engineering  should  be  required  to  complete  the  distribution  to  the  field. 
This  "percentage"  is  bound  by  a  lower  absolute  value  of  time  required  based  on  physical  limitations 
(e.g.,  prom  burning,  technical  order  generation)  of  the  distribution  process. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(CC)  -  020 

QUESTION:  The  configuration  control  responsibility  for  integrating  computer  resources  into  the 
system  has  remained  centralized  throughout  the  life  of  the  system. 

ACTIVITY(IES):  All 

EXPLANATION:  Although  organizational  elements  (e.g.,  HQ  ACC,  HQ  AFSPACECOM)  may  have 
configuration  control  responsibilities  for  separate  elements  of  the  system  (e.g.,  software,  hardware), 
there  should  be  a  centralized  control  point  for  all  decisions  (perhaps  a  set  of  configuration  control 
boards).  As  the  software  is  passed  from  the  development  contractor  to  the  operational  support 
activity,  the  configuration  control  responsibilities  are  passed  from  the  centralized  development 
configuration  control  to  the  centralized  support  configuration  control,  with  appropriate  planning  and 
attention  to  the  actual  transfer  of  responsibility. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE; 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(CC)  -  021 

QUESTION:  The  configuration  control  requirements  for  postdeployment  support  are  adequately 
addressed  in  the  CRLCMP. 

ACTIVITY(IES):  Operational  Support 

EXPLANATION:  The  Computer  Resources  Life  Cycle  Management  Plan  (CRLCMP)  is  the  key 
planning  document  for  operational  support  configuration  management.  The  CRLCMP  is  intended  to 
be  a  living  document,  evolving  to  provide  a  current  view  of  the  configuration  management  features 
along  with  the  evolution  of  the  system.  The  CRLCMP  (first  version)  is  required  early  in  the  hfe 
cycle,  usually  in  concept  exploration.  Key  to  adequacy  is  the  compatibility  of  the  operational  support 
configuration  control  and  the  development  contractor  configuration  control  procedures  and 
automated  tool  support. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4,3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(CC)  -  022 

QUESTION:  The  operational  support  configuration  control  boards  are  adequately  defined  to  handle 
software  changes. 

ACTIVITY(IES):  Operational  Support 

EXPLANATION:  The  Computer  Resources  Life  Cycle  Management  Plan  (CRLCMP)  is  the  key 
planning  document  for  operational  support  configuration  management.  The  CRLCMP  is  intended  to 
be  a  living  document,  evolving  to  provide  a  current  view  of  the  configuration  management  features 
along  with  the  evolution  of  the  system.  The  configuration  control  boards  along  with  specific  board 
responsibilities  should  be  defined  in  the  CRLCMP.  It  is  not  enough  to  indicate  that  a  given  directive, 
regulation,  standard,  or  guideline  will  be  followed.  Specific  detail  as  to  the  board  function, 
relationship  to  the  organizational  structure,  interface  responsibilities,  and  so  forth  should  be 
included  in  the  operational  support  configuration  management  plan  and  procedures. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6,5,4,3,2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(CC)  -  023 

QUESTION:  The  automated  support  tools  for  postdeployment  support  of  configuration  control  are 
adequately  addressed  in  the  CRLCMP. 

ACTIV IT Y(IES):  Operational  Support 

EXPLANATION:  The  Computer  Resources  Life  Cycle  Management  Plan  (CRLCMP)  is  the  key 
planning  document  for  operational  support  configuration  management.  The  CRLCMP  is  intended  to 
be  a  living  document,  evolving  to  provide  a  current  view  of  the  configuration  management  features 
along  with  the  evolution  of  the  system.  The  use  of  automated  support  tools  during  development  and 
transition  of  those  tools  to  use  dtiring  postdeployment  software  support  is  an  important  considera¬ 
tion  for  the  overall  enhancement  of  software  supportability.  The  lack  of  such  tools  to  assist  in  the 
configuration  control  of  the  various  baselines  should  be  considered  a  serious  deficiency. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5,4, 3,2,1  =  COMPLETELY  DISAGREE) 
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SOFTWARE  CONFIGURATION  MANAGEMENT  STATUS  ACCOUNTING 


The  questions  SCM(SA)-001  through  SCM(SA)-015  address  issues  of  software  configuration 
status  accounting  for  the  procurement,  development  contractor,  and  operational  support  activities. 
Configuration  status  accounting  is  the  process  of  keeping  track  of  the  configuration  identification 
and  its  changes  and  reporting  this  information  to  management.  Two  types  of  documents  are  pro¬ 
duced  by  configuration  status  accounting:  (1)  Software  Configuration  Index:  defines  the  current 
approved  configuration  of  an  item  in  terms  of  its  elements  or  identification  documents  and  its 
approved  changes,  and  (2)  Change  Status  Reports:  for  deficiency  and  modification  changes  to  a 
configured  item. 

These  configuration  status  accounting  documents  provide  all  activities  with  visibility  and 
traceability  of  baseline  configurations  and  their  changes.  Coordination  of  activities  and  decisions 
regarding  these  activities  such  as  scheduled  reviews,  audits,  tests,  use  of  test  resources, 
requirements  for  budget  adjustments  to  the  contract,  and  so  forth  are  based  on  configuration  status 
accounting  information. 
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Question  Number  SCM(SA)  -  001 

QUESTION:  The  procurement  policy,  standards,  and  conventions  applied  to  the  configuration 
status  accounting  of  software  configuration  items  are  adequate. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  Documentation  for  describing  and  reporting  the  status  of  computer  software 
configured  items  is  described  in  DoD  and  Air  Force  directives,  regulations,  and  guidelines,  including 
AFR  57-4,  AFR  800-14,  and  DoDD  5010.21.  It  is  the  responsibility  of  the  procurement  activity  to 
ensure  that  proper  policy,  standards,  and  conventions  are  required  for  the  configuration  status 
accounting  of  configured  items.  Contractor  compliance  documents  that  could  be  required  include 
MIL-STD-480A,  MIL-STD-482,  MIL-STD-483A,  MIL-STD-490A,  and  DoD-STD-2167A. 

GLOSSARY: 

Software  Configuration  Items.  Software  elements  that  are  designated  for  configuration  status 
accounting  by  the  contractual  requirements. 

Configuration  Status  Accounting.  The  means  through  which  actions  affecting  CSCIs  are 
recorded  and  reported  to  program  and  functional  managers. 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(SA)  -  002 

QUESTION:  The  procurement  activity  has  implemented  adequate  software  configuration  status 
accounting,  based  on  regulations,  to  report  the  functional  and  physical  characteristics  of  all  CSCIs. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  procurement  program  manager  is  responsible  for  implementing  a  configura¬ 
tion  management  program  that  will  identify,  documept,  and  control  the  functional  and  physical 
characteristics  of  all  CSCIs  under  development.  Primary  planning  document  is  the  Program  Man¬ 
agement  Plan  (PMP).  Other  activities  include  coordinating  requirements  with  using  and  supporting 
agencies,  reviewing  contractor  plans,  auditing  contractor  implementation  of  plans,  ensuring  con¬ 
figuration  identifications  for  all  CSCIs  are  properly  documented,  controlling  engineering  changes  to 
baselines,  controlling  distribution  of  changes,  and  preparing  for  transfer  to  the  operational  support 
activity. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(SA)  -  003 

QUESTION:  The  procurement  configuration  management  planning  documents  contain  sufficient 
guidance  for  configuration  status  accounting. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  major  planning  documents  for  procurement  are  the  Program  Management 
Plan  (PMP),  the  Request  for  Proposal  (RFP)/Statement  of  Work  (SOW),  the  Contract  Data 
Requirements  List  (CDRL),  and  the  Computer  Resources  Life  Cycle  Management  Plan  or  CRLCMP. 
APR  800-14  calls  for  the  inclusion  of  configuration  management  concepts  in  the  PMP  (including 
specification  and  interfaces).  The  RFP/SOW  defines  the  exact  scope  of  the  development  contractor's 
configuration  status  accounting  responsibilities.  The  CDRL  identifies  all  deliverable  data  items 
including  CSCIs  that  the  development  contractor  must  dehver  and  control.  The  CRLCMP  is  to 
include  CSCI  configuration  baseline  reporting  procedures  to  account  for  the  implementation  of 
changes.  Detailed  procedures  should  also  be  in  the  CRLCMP. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(SA)  -  004 

QUESTION:  The  procurement  activity  configuration  status  accounting  procedures  are  adequate. 
ACTIVITY(IES):  Procurement 

EXPLANATION:  The  procurement  activity  configuration  status  accounting  procedures  include 
procedures  to  report  baseline  configuration  identification  and  change  status,  contract  management 
modifications,  specification  status  and  changes,  ECPs  and  SCNs,  and  any  other  documents  that 
record  the  software  history  of  development  and  support.  This  history  will  to  he  transferred  to  the 
operational  support  activity.  This  history  provides  traceability  to  the  configuration  management 
process  and  the  resulting  software  products.  Use  of  automated  support  tools  should  aid  the 
effectiveness  and  efficiency  of  the  configuration  status  accounting  procedures. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6,5,4,3,2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(SA)  -  005 

QUESTION:  The  development  contractor  internal  configuration  status  accounting  procedures  are 
adequate. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  Procedures  should  be  documented  in  a  required  software  configuration  manage¬ 
ment  plan.  Procedures  include  how  information  on  status  is  to  be  collected,  verified,  stored, 
processed,  and  reported  and  the  identification  of  the  periodic  reports  to  be  provided  and  their 
distribution.  Information  to  be  maintained  includes  status  of  specifications  and  proposed  changes, 
reports  of  approved  changes,  status  of  product  versions  or  revisions,  reports  of  the  implementation  of 
installed  updates  or  releases,  and  status  of  procurement  supplied  property. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 


198 


AFOTECPAM  99-102  Volume  2  Attachment  1  1  August  1994 


QUESTION  DATA  SHEET 


Question  Number  SCM(SA)  -  006 

QUESTION:  The  development  contractor  configuration  status  accounting  has  an  adequate  interface 
with  the  procurement  activity  configuration  status  accounting. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  AFR  800-14  requires  computer  program  configuration  management  to  be 
integrated  into  the  overall  system  configuration  management  and  across  all  cognizant  organization 
elements.  The  status  accounting  interface  between  procurement  and  development  contractor  is  the 
basis  for  reporting  all  significant  baseline  product  actions  and  the  current  state  of  those  actions. 
Early  resolution  of  problems  is  a  direct  function  of  how  accurately,  concisely,  and  efficiently  such 
status  accounting  information  is  presented.  The  primary  changes  that  require  interface  status 
reports  are  Class  I  changes.  MIL-STD-483A  is  the  primary  development  contractor  compliance 
regulation. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(SA)  -  007 

QUESTION:  The  development  contractor  configuration  status  accounting  procedures  can  be 
transitioned  to  or  are  compatible  with  the  operational  support  activity  planned  configuration  status 
accounting  procedures. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  CRLCMP  describes  the  operational  support  planned  configuration  status 
accounting  procedures,  usually  in  accordance  with  DoD-STD-480A  and  MIL-STD-483A.  The 
operational  support  activity's  internal  configuration  status  accoimting  procedures  should  be 
documented  in  a  Software  Configuration  Management  Plan  or  an  associated  set  of  procedures. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(SA)  -  008 

QUESTION:  The  development  contractor  automated  support  tools  for  configuration  status 

accounting  of  baselines  and  internal  development  identifications  are  adequate. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  automated  support  tools  might  include  a  control  library,  automated 
procedures  for  form  generation  and  retrieval,  automatic  traceabihty  for  version  control,  implemented 
changes,  and  outstanding  problem  reports.  Traceability  of  requirements,  design,  code,  and  test 
elements  is  important  for  keeping  track  of  precise  configuration  identification  of  baseline  data.  The 
use  of  automated  tools  is  essential  for  complex  software  systems. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6,5,4,3,2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(SA)  -  009 

QUESTION:  The  development  contractor  software  configuration  status  accounting  forms  are 
adequate. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  status  accounting  forms  must  provide  adequate  information  to  track  internal 
development  baselines  as  well  as  the  Functional,  Allocated,  and  Product  baselines.  MIL-STD-482A 
contains  configuration  status  accounting  data  elements  and  related  features.  Typical  government 
forms  include  ECP,  SCN,  configuration  control  board  directive,  time  compliance  technical  order,  defi¬ 
ciency  report,  and  change/modification  report.  Internal  status  accounting  forms  must  be  adequate  to 
track  necessary  status  reporting  such  as  problem  analysis,  solution,  change  implementation,  and 
closure.  In  addition,  general  reporting  documents  such  as  a  product  status  report,  open  software 
problems  report,  and  deliverable  document  status  report  must  be  maintained  in  order  that 
contractually  required  status  information  can  be  adequately  derived  and  justified. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6,5,4,3,2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(SA)  -  010 

QUESTION:  Subcontractor  configuration  item  configuration  status  accounting  procedures  are 
monitored  by  the  development  contractor. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  If  there  is  a  subcontractor,  it  will  be  necessary  that  the  development  contractor 
require  configuration  status  accounting  practices  similar  to  those  required  by  the  procurement 
activity.  If  this  is  not  done,  then  the  development  contractor  will  be  required  to  retrofit  the 
configuration  status  accounting  scheme  of  the  subcontractor.  The  configuration  status  accounting 
identification  practices  of  the  subcontractor  must  be  carefully  monitored  to  ensure  compatibility. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

A/6:  No  subcontractors  are  involved  with  producing  software  configuration  items  for  the 

development  contractor. 

RESPONSE  RATIONALE: 


RESPONSE  SCOREi 

(COMPLETELY  AGREE  =  6, 5, 4,3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(SA)  -  Oil 

QUESTION:  Status  of  software  configuration  items  that  implement  safety  provisions  is  adequately 
reported. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  Configured  items  that  implement  safety  provisions  are  frequently  controlled  by 
software.  Status  of  this  software  must  be  adequately  monitored  and  reported  as  affecting  safety. 
Safety  provisions  are  closely  related  to  the  reliability  of  mission  critical  components,  safety  of  mission 
personnel,  nuclear  effects,  and  so  forth. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

A/6:  There  is  no  requirement  for  safety  provisions  to  be  implemented  or  controlled  software. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5,4, 3,2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(SA)  -  012 

QUESTION:  Status  of  software  configured  items  that  implement  computer/communications 

security  provisions  is  adequately  reported. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  Software  that  implements  computer/communication  security  is  particularly 
important.  Any  such  software  items  must  be  adequately  controlled  as  part  of  the  trusted  computer 
base.  If  the  configured  software  items  are  themselves  classified,  then  appropriate  security  labels 
must  be  attached  according  to  Air  Force  labeling  requirements  and  access  control  of  such  items  must 
be  enforced.  Status  information  on  such  software  must  be  reported. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

A/6:  There  is  no  requirement  for  security  provisions  to  be  implemented  in  software. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(SA)  -  013 

QUESTION:  The  configuration  status  accounting  requirements  for  postdeployment  support  are 
adequately  addressed  in  the  CRLCMP. 

ACTIVITY(IES):  Operational  Support 

EXPLANATION:  The  Computer  Resources  Life  Cycle  Management  Plan  (CRLCMP)  is  the  key 
planning  document  for  operational  support  configuration  management.  The  CRLCMP  is  intended  to 
be  a  living  document,  evolving  to  provide  a  current  view  of  the  configuration  management  features 
along  with  the  evolution  of  the  system.  The  CRLCMP  (first  version)  is  required  early  in  the  life 
cycle,  at  least  prior  to  engineering  and  manufacture  development.  Key  to  adequacy  is  the 
compatibility  of  the  operational  support  configuration  status  accounting  and  the  development 
contractor  configuration  status  accounting. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE; 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(SA)  -  014 

QUESTION:  The  operational  support  configuration  status  accounting  procedures  are  adequately 
defined  to  handle  software  change  reporting  requirements. 

ACTIVITY(IES):  Operational  Support 

EXPLANATION:  The  Computer  Resources  Life  Cycle  Management  Plan  (CRLCMP)  is  the  key 
planning  document  for  operational  support  configuration  management.  The  CRLCMP  is  intended  to 
be  a  living  document,  evolving  to  provide  a  current  view  of  the  configuration  management  features 
along  with  the  evolution  of  the  system.  The  configuration  status  accounting  procedures  together 
with  specific  responsibilities  should  be  defined  in  the  CRLCMP.  It  is  not  enough  to  indicate  that  a 
given  directive,  regulation,  standard,  or  guideline  will  be  followed.  Specific  details  as  to  the  format 
and  content  of  status  reports,  organizational  structure,  interface  responsibilities,  and  so  forth  should 
be  included  in  the  operational  support  Software  Configuration  Management  Plan. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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Question  Number  SCM(SA)  -  015 

QUESTION:  The  automated  support  tools  for  postdeployment  support  of  configuration  status 
accounting  are  adequately  addressed  in  the  CRLCMP. 

ACTIVITY(IES):  Operational  Support 

EXPLANATION:  The  Computer  Resources  Life  Cycle  Management  Plan  (CRLCMP)  is  the  key 
planning  document  for  operational  support  configuration  management.  The  CRLCMP  is  intended  to 
be  a  living  document,  evolving  to  provide  a  current  view  of  the  configuration  management  features 
along  with  the  evolution  of  the  system.  The  use  of  automated  support  tools  during  development  and 
tremsition  of  those  tools  to  use  during  postdeployment  support  is  an  importemt  consideration  for  the 
overall  enhancement  of  software  supportability.  The  lack  of  such  tools  to  manage  the  configuration 
status  accounting  of  the  various  baselines  should  be  considered  a  deficiency. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

(COMPLETELY  AGREE  =  6, 5, 4, 3, 2,1  =  COMPLETELY  DISAGREE) 
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SOFTWARE  CONFIGURATION  MANAGEMENT  AUDIT/REVIEW 


The  questions  SCM(AR)-001  through  SCM(AR)-013  address  issues  of  software  configuration 
Audit/Review  for  the  procurement,  development  contractor,  and  operational  support  activities. 
Software  Configuration  Audit/Review  is  conducted  to  verify  that  a  completed  software  product 
satisfies  requirements.  Procurement  conducts  official  contractual  configuration-oriented  audits  and 
reviews.  Portions  of  the  development  reviews  (PDR,  CDR,  Test  Readiness  Review  (TRR))  are 
devoted  to  configuration-oriented  review  of  production  products  as  identified  in  developmental 
baselines.  The  major  configuration  audits  for  procurement  are: 

(1)  Functional  Configuration  Audit  (FCA) 

(2)  Physical  Configuration  Audit  (PCA) 

A  Formal  Qualification  Review  (FQR)  for  each  CSCI  constitutes  a  configuration  audit  if  it  is 
included  as  part  of  the  CSCI  Configuration  Index.  In  this  case,  procurement  and  perhaps 
operational  support  representatives  review  the  product  specifications,  Preliminary  Qualification  Test 
(PQT)  data,  and  Formal  Qualification  Test  (FQT)  data  to  certify  that  the  CSCI  is  qualified  for  its 
intended  application.  The  FQR  follows  the  FCA  and  PCA. 

The  operational  support  activity  has  a  responsibility  to  monitor  procurement  and  development 
contractor  audits  and  reviews.  This  monitor  information  is  integrated  into  the  operational  support 
plans.  The  operational  support  configuration  audit/review  is  similar  to  the  procurement  for  the 
formal  baselines  and  like  the  development  contractor  for  the  internal  audits  and  reviews. 
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QUESTION  DATA  SHEET 


Question  Number  SCM(AR)  -  001 

QUESTION:  The  procurement  policy,  standards,  and  conventions  applied  to  the  audit  and  review  of 
software  configuration  items  are  adequate. 

ACTIVITY(IES);  Procurement 

EXPLANATION:  Audit/review  of  computer  software  configured  items  is  described  in  directive, 
regulations,  and  guideline,  including  AFR  57-4,  DoDD  5000.29,  AFR  800-14,  and  DoDD  5010.21.  It 
is  the  responsibility  of  the  procurement  activity  to  ensure  that  proper  policy,  standards,  and  conven¬ 
tions  are  required  for  the  audit/review  of  configured  item.  Contractor  compliance  documents  that 
could  be  required  include  MIL-STD-480A,  MIL-STD-482,  MIL-STD-483A,  MIL-STD-490A,  MIL-STD- 
1521B,  and  DoD-STD-2167A. 

GLOSSARY: 

Software  Configuration  Items.  Software  elements  that  are  designated  for  Configuration 
Audit/Review  by  the  contractual  requirements. 

Audit/Review.  The  process  of  informal  and  formal  verification  that  a  particular  product  has 
satisfied  a  specified  set  of  requirements. 

RESPONSE  INSTRUCTIONS: 


RESPONSE  RATIONALE: 
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QUESTION  DATA  SHEET 


Question  Number  SCM(AR)  -  002 

QUESTION:  The  procurement  activity  has  implemented  adequate  software  configuration  audit/ 
review  based  on  regulations  to  control  the  functional  and  physical  characteristics  of  all  CSCIs. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  procurement  program  manager  is  responsible  for  implementing  a  configura¬ 
tion  management  program  that  will  identify,  document,  and  control  the  functional  and  physical 
characteristics  of  all  CSCIs  under  development.  Primary  planning  document  is  the  Program 
Management  Plan  (PMP).  Other  activities  include  coordinating  requirements  with  using  and 
supporting  agencies,  reviewing  contractor  plans,  auditing  contractor  implementation  of  plans, 
ensuring  configuration  identifications  for  all  CSCIs  are  properly  documented,  controlling  engineering 
changes  to  baselines,  providing  interface  control  for  distribution  of  changes,  and  preparing  for 
transfer  to  the  operational  support  activity. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 


RESPONSE  RATIONALE: 
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QUESTION  DATA  SHEET 


Question  Number  SCM(AE)  -  003 

QUESTION:  The  procurement  configuration  management  planning  documents  contain  sufficient 
guidance  for  configuration  audit/review. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  major  planning  documents  for  procurement  are  the  Program  Management 
Plan  (PMP),  the  Request  for  Proposal  (RFP)/Statement  of  Work  (SOW),  the  Contract  Data 
Requirements  List  (CDRL),  and  the  Computer  Resources  Life  Cycle  Management  Plan  (CRLCMP). 
AFR  800-14  calls  for  the  inclusion  of  configuration  management  concepts  in  the  PMP  including 
specification  and  interfaces.  The  RFP/SOW  defines  the  exact  scope  of  the  development  contractor's 
configuration  audit/review  responsibilities.  The  CDRL  identifies  all  deliverable  data  items  including 
CSCIs  that  the  development  contractor  must  deliver  and  control.  The  CRLCMP  is  to  include 
assignment  of  configuration  audit/review  responsibilities  during  postdeployment  with  detailed 
procedures. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 


RESPONSE  RATIONALE: 
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QUESTION  DATA  SHEET 


Question  Number  SCM(AR)  -  004 

QUESTION:  The  conduct  of  formal  reviews  and  audits  follows  a  format  based  on  the  checklists  from 
MIL-STD-1521B,  appropriately  tailored  for  the  specific  software  audit/review. 

ACTIVITY(IES):  Procurement 

EXPLANATION:  The  MIL-STD-1521B  is  the  compliance  document  for  development  contractor 
audits  and  reviews.  It  is  procurement's  responsibility  to  provide  the  guidance  for  standards,  regula¬ 
tions,  and  tailoring  guidance  that  is  required.  It  is  the  development  contractor's  responsibility  to 
follow  the  requirements.  There  are  related  configuration  audits  and  evaluation  checklists  (e.g.,  EGA 
and  PGA  preparation  checklists  in  MIL-STD-1521B,  EGP  preparation  checklists  in  DoD-STD-480A 
and  modified  by  MIL-STD-483A,  Gomputer  Resource  Manager's  Ghecklist  based  on  APR  800-14). 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 


RESPONSE  RATIONALE: 
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QUESTION  DATA  SHEET 


Question  Number  SCM(AR)  -  005 

QUESTION:  The  software  product  acceptance  requirements  are  adequate. 

ACTIV IT Y (lES):  Procurement 

EXPLANATION:  The  software  product  acceptance  criteria  should  be  clearly  documented.  The 
acceptance  tests,  demonstrations,  DT&E,  OT&E,  quahfication  tests,  audits,  and  reviews  all  form  a 
part  of  these  acceptance  requirements.  The  procurement  activity  has  the  responsibility  to  make  sure 
such  acceptance  requirements  are  cost  effective,  functionally  adequate,  and  specified  from  the  time  of 
the  RFP/SOW/CDRL.  Frequent  modification  to  the  original  requirements  indicates  a  lack  of 
imderstanding  concerning  the  original  system  specifications.  This  is  likely  to  result  in  a  less  mature 
system  to  be  supported. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 
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QUESTION  DATA  SHEET 


Question  Number  SCM(AR)  -  006 

QUESTION:  The  development  contractor  internal  configuration  audit/review  process  facilitates  the 
development  of  high  quality  production  software. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  internal  development  contractor  procedures  for  audit  and  review  can  be  an 
important  part  of  the  process  to  build  in  software  supportability  characteristics  in  the  software 
products.  This  shows  up  on  hoth  the  transition  of  life  cycle  processes  and  in  the  transition  of  the  soft¬ 
ware  product  baseline.  The  internal  audit/review  process  also  tends  to  reflect  how  successful  the 
formal  contractual  audit/reviews  will  be. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 


RESPONSE  RATIONALE: 
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QUESTION  DATA  SHEET 


Question  Number  SCM(AR)  -  007 

QUESTION:  Configuration  audit/review  interfaces  among  procurement,  development  contractor, 
and  operational  support  activities  are  adequate. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  The  activities  require  information  from  all  levels  of  the  audit/review  process  in 
order  to  properly  plan  for  specific  activity  resources,  funding  levels,  resolution  of  problems,  and  so 
forth.  An  interface  control  working  group  is  an  appropriate  medium  for  coordinating  schedule, 
responsibilities,  contractual  aspects,  and  results  of  the  audit/reviews. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 


RESPONSE  RATIONALE: 
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QUESTION  DATA  SHEET 


Question  Number  SCM(AR)  -  008 

QUESTION:  The  development  contractor  configuration  management  tool  support  facilitates  the 
audit/review  of  the  process  by  which  changes  are  incorporated  into  configuration  identifications. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  It  is  required  to  audit/review  all  changes  that  have  heen  incorporated  into  a 
configuration  identification.  It  greatly  facilitates  the  audit/review  process  if  the  change  process  is 
automated  and  tool  support  is  available  to  indicate  the  configuration  identification  with  and  without 
the  incorporated  changes.  Configuration  identification  compactor  tools  can  indicate  which  elements 
of  the  configuration  identification  have  been  changed  as  a  confirmation  of  the  incorporated  changes. 
The  availability  of  such  automated  tool  support  greatly  facilitates  the  efficiency  and  accuracy  of  the 
audit  and  review  activity. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 


RESPONSE  RATIONALE: 
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QUESTION  DATA  SHEET 


Question  Number  SCM(AR)  -  009 

QUESTION:  Subcontractor  configuration  item  audit/review  practices  are  monitored  by  the  develop¬ 
ment  contractor. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  If  there  is  a  subcontractor,  it  will  be  necessary  that  the  development  contractor 
require  configuration  audit/review  practices  similar  to  those  required  by  the  procurement  activity.  If 
this  is  not  done,  then  the  development  contractor  will  be  required  to  retrofit  the  audit/review  scheme 
of  the  subcontractor.  The  audit/review  practices  of  the  subcontractor  must  be  carefully  monitored  to 
ensure  compatibility. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

A/6:  No  subcontractors  are  involved  with  producing  software  configuration  items  for  the 

development  contractors. 


RESPONSE  RATIONALE: 
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QUESTION  DATA  SHEET 


Question  Number  SCM(AR)  -  010 

QUESTION:  Configured  items  that  implement  safety  provisions  are  adequately  audited  and 
reviewed. 

ACTIVITY(IES):  Development  Contractor 

EXPLANATION:  Configured  items  that  implement  safety  provisions  are  frequently  controlled  by 
software.  This  software  must  be  adequately  identified  as  affecting  safety.  Safety  provisions  are 
closely  related  to  the  reliability  of  mission  critical  components  and  safety  of  mission  personnel, 
nuclear  effects,  and  so  forth. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

A/6:  There  is  no  requirement  for  safety  provisions  to  be  implemented  or  controlled  by  software. 


RESPONSE  RATIONALE: 
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QUESTION  DATA  SHEET 


Question  Number  SCM(AR)  -  Oil 

QUESTION:  Software  configured  items  that  implement  computer/communications  security  provi¬ 
sions  are  adequately  audited  and  reviewed, 

ACTrVlTY(IES):  Development  Contractor 

EXPLANATION:  Software  that  implements  computer/communication  security  is  particularly 
important.  Any  such  software  items  must  be  adequately  audited/reviewed  as  part  of  the  trusted  data 
base.  If  the  configured  software  items  are  themselves  classified,  then  appropriate  security  labels 
must  be  attached  according  to  Air  Force  labeling  requirements.  Adequacy  of  such  labeling  proce¬ 
dures  should  be  audited/reviewed. 

GLOSSARY: 

Security  Provisions.  The  totality  of  threats,  vulnerabilities,  and  protection  mechanisms  involved 
with  determining  whether  computer/communications  assets  can  be  compromised  through  data, 
process,  or  abuse  violations.  Security  provisions  exist  across  the  administrative,  system,  and  facility 
categories. 

RESPONSE  INSTRUCTIONS: 

A/6:  There  is  no  requirement  for  security  provisions  to  be  implemented  in  software. 


RESPONSE  RATIONALE: 


220 


AFOTECPAM  99-102  Volume  2  Attachment  1  1  August  1994 


QUESTION  DATA  SHEET 


Question  Number  SCM(AR)  -  012 

QUESTION:  The  software  configuration  audit/review  requirements  for  postdeployment  support  are 
adequately  addressed  in  the  CRLCMP. 

ACTIVITY(IES):  Operational  Support 

EXPLANATION:  The  Computer  Resources  Life  Cycle  Management  Plan  (CRLCMP)  is  the  key 
planning  document  for  operational  support  configuration  management.  The  CRLCMP  is  intended  to 
be  a  living  document,  evolving  to  provide  a  current  view  of  the  configuration  management  features 
along  with  the  evolution  of  the  system.  The  CRLCMP  (first  version)  is  required  early  in  the  life 
cycle,  at  least  prior  to  engineering  and  manufacture  development. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 
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QUESTION  DATA  SHEET 


Question  Number  SCM(AR)  -  013 

QUESTION:  The  automated  support  tools  for  postdeployment  support  of  configuration  audit/review 
are  adequately  addressed  in  the  CRLCMP. 

ACTIVITY(IES):  Operational  Support 

EXPLANATION:  The  Computer  Resources  Life  Cycle  Management  Plan  (CRLCMP)  is  the  key 
planning  document  for  operational  support  configuration  management.  The  CRLCMP  is  intended  to 
be  a  living  document,  evolving  to  provide  a  current  view  of  the  configuration  management  features 
along  with  the  evolution  of  the  system.  The  use  of  automated  support  tools  during  development  and 
transition  of  those  tools  for  use  during  postdeployment  support  is  an  important  consideration  for  the 
overall  enhancement  of  software  supportability.  The  lack  of  such  tools  to  assist  in  the  audit  and 
review  of  the  various  baselines  should  be  considered  a  serious  deficiency. 

GLOSSARY: 

RESPONSE  INSTRUCTIONS: 

RESPONSE  RATIONALE: 
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SUMMARY  LIST  OF  QUESTIONS 


This  attachment  contains  a  summary  list  of  all  Software  Life  Cycle  Process  questions.  The  questions 
are  listed  in  the  order  in  which  they  appear  in  attachment  1. 

SOFTWARE  PROJECT  MANAGEMENT  PLANNING 


SPM(PL)  -  001:  QUESTION:  Planning  for  computer  resources  has  been  adequate  with  respect  to 
acquisition,  development,  logistics,  and  training. 

SPM(PL)-002:  QUESTION:  Procurement  planning  for  computer  resources  has  been  consistent 
with  the  system  development  and  acquisition  plan. 

SPM(PL)  -  003:  QUESTION:  Planning  for  computer  resources  has  been  based  upon  an  acquisition 
schedule  with  adequately  specified  milestones. 

SPM(PL)  -  004:  QUESTION:  Computer  resources  have  been  adequately  addressed  as  major  consid¬ 
erations  at  procurement  reviews,  audits,  and  management  evaluations. 

SPM(PL)  -  005:  QUESTION:  Planned  computer  resources  have  been  analyzed  adequately  by  pro¬ 
curement  to  ensure  conformance  with  stated  operational  and  support  requirements. 

SPM(PL)  -  006:  QUESTION:  Procurement  planning  software  quality  attributes  have  been  ade¬ 
quately  emphasized  throughout  the  software  life  cycle  acquisition. 

SPM(PL)  -  007:  QUESTION:  Margins  for  reserve  computer  resource  capacity  to  provide  for  later 
product  improvements  are  adequate. 

SPM(PL)  -  008:  QUESTION:  Acceptable  techniques  have  been  used  to  estimate  and  monitor  soft¬ 
ware  costs  throughout  the  system  life  cycle. 

SPM(PL)  -  009:  QUESTION:  The  CRLCMP  contains  adequate  specifications  of  the  acquisition 
requirements  for  computer  resources. 

SPM(PL)  -  010:  QUESTION:  The  CRLCMP  adequately  addresses  the  responsibihties  and  proce¬ 
dures  to  ensure  proper  software  configuration  management  throughout  the  system  life  cycle. 

SPM(PL)  -  Oil:  QUESTION:  The  project  management  responsibility  for  integrating  computer 
resources  into  a  system  has  remained  centralized  throughout  the  life  of  the  system. 

SPM(PL)  -  012:  QUESTION:  The  CRWG  organization  has  been  adequate  throughout  the  system  life 
cycle. 

SPM(PL)  -  013:  QUESTION:  The  CRWG  has  had  clearly  specified  responsibilities  and  appropriate 
authority  to  implement  those  responsibilities  throughout  the  system  life  cycle. 

SPM(PL)  -  014:  QUESTION:  The  CRWG  has  properly  ensured  that  computer  resources  comply  with 
established  policy,  procedures,  plans,  and  standards. 
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SPM(PL)  -  015:  QUESTION:  Software  quality  assessment  procedures  have  been  adequately  defined 
to  meet  management  policies  and  appropriate  regulations,  conform  to  standards,  and  meet  perform¬ 
ance  and  quality  requirements  throughout  the  system  life  cycle. 

SPM(PL)  -  016:  QUESTION:  Planning  for  DT&E  of  computer  resources  has  been  adequate  through¬ 
out  the  system  life  cycle. 

SPM(PL)  -  017:  QUESTION:  Planning  for  OT&E  of  computer  resources  has  been  adequate  through¬ 
out  the  system  life  cycle. 

SPM(PL)  -  018:  QUESTION:  Software  standards  have  been  adequately  specified  throughout  the 
software  life  cycle. 

SPM(PL)  -  019:  QUESTION:  The  planning  for  organic  and/or  contractor  support  during  post¬ 
deployment  software  support  has  been  adequate. 

SPM(PL)  -  020:  QUESTION:  Contractual  documents  have  explicitly  established  government  rights 
to  all  computer  resources  required  to  develop,  operate,  simulate,  test,  and  support  the  software. 

SPM(PL)  -  021:  QUESTION:  Planning  for  risk  analysis  to  identify  areas  of  computer  resource  risk 
has  been  adequate. 

SPM(PL)  -  022:  QUESTION:  A  mission/function  matrix  (or  equivalent)  clearly  identifies  primary 
functional  capabilities  to  be  implemented  by  the  software. 

SPM(PL)  -  023:  QUESTION:  Planning  for  interoperability  with  other  systems  has  been  adequately 
addressed. 

SPM(PL)  -  024:  QUESTION:  Prior  to  each  system  milestone,  interservicing  potential  and  life  cycle 
cost  implications  of  software  support  options  have  been  appropriately  addressed. 

SPM(PL)  -  025:  QUESTION:  The  procurement  and  operational  support  planning  documents  have 
been  adequately  updated  as  living  documents  throughout  the  system  life  cycle. 

SPM(PL)  -  026:  QUESTION:  The  principles  and  methodologies  provided  in  the  regulations  have 
been  appropriately  incorporated  into  the  software  test  and  evaluation  plans. 

SPM(PL)-027:  QUESTION:  Planning  for  systematic,  quantitative,  and  objectively  reportable 
software  tests  has  been  adequate. 

SPM(PL)  -  028:  QUESTION:  Planning  for  sharing  of  software  test  results  across  life  cycle  phases 
and  among  test  organizations  has  been  adequate. 

SPM(PL)  -  029:  QUESTION:  Tracking  of  computer  resource  utilization  has  been  adequately 
planned. 

SPM(PL)  -  030:  QUESTION:  The  project  software  budget/cost  variance  (budgeted  -  actual)  appears 
to  be  reasonable. 

SPM(PL)  -  031:  QUESTION:  The  project  software  schedule/cost  variance  (consumed  -  scheduled) 
appears  to  be  reasonable. 

SPM(PL)  -  032:  QUESTION:  The  cost  and  schedule  contractual  reporting  requirements  appear  to  be 
adequate. 
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SOFTWARE  PROJECT  MANAGEMENT  ORGANIZATION  STRUCTURE 


SPM(OS)  -  001:  QUESTION:  The  software  requirements  have  been  adequately  allocated  to 
elements  of  a  Work  Breakdown  Structure  (WBS). 

SPM(OS)  -  002:  QUESTION:  The  software-related  tasks  are  clearly  identified  in  the  WBS. 

SPM(OS)  -  003:  QUESTION:  The  key  project  personnel  and  their  assignments  in  relation  to  the 
WBS  software  related  tasks  are  clearly  identified. 

SPM(OS)  -  004:  QUESTION:  The  coordination  of  modifications  to  the  WBS  among  all  activities  has 
been  adequate. 

SPM(OS)  -  005:  QUESTION:  The  procurement  personnel  staffing  has  had  continuity  throughout 
the  software  life  cycle  phases. 

SPM(OS)  -  006:  QUESTION:  The  ratio  of  experienced  procurement  project  personnel  to  the  total 
number  of  project  personnel  has  been  adequate  throughout  the  software  life  cycle  phases. 

SPM(OS)  -  007:  QUESTION:  The  number  of  procurement  personnel  has  been  adequate  throughout 
the  software  life  cycle  phases. 

SPM(OS)  -  008:  QUESTION:  The  development  contractor  personnel  staffing  has  had  continuity 
throughout  the  software  life  cycle  phases. 

SPM(OS)  -  009:  QUESTION:  The  ratio  of  experienced  development  contractor  project  personnel  to 
the  total  number  of  project  personnel  has  been  adequate  throughout  the  software  life  cycle  phases. 

SPM(OS)  -  010:  QUESTION:  The  number  of  development  contractor  personnel  has  been  adequate 
throughout  the  software  life  cycle  phases. 

SPM(OS)-011:  QUESTION:  The  operational  support  persoimel  staffing  has  had  continuity 
throughout  the  software  life  cycle  phases. 

SPM(OS)  -  012:  QUESTION:  The  ratio  of  experienced  operational  support  project  personnel  to  the 
total  number  of  project  personnel  has  been  adequate  throughout  the  software  life  cycle  phases. 

SPM(OS)  -  013:  QUESTION:  The  number  of  operational  support  personnel  has  been  adequate 
throughout  the  software  life  cycle  phases. 

SPM(OS)-O14:  QUESTION:  The  internal  interfaces  among  procurement  organization  elements 
have  been  adequate. 

SPM(OS)  -  015:  QUESTION:  The  internal  interfaces  among  development  contractor  organization 
elements  have  been  adequate. 

SPM(OS)-O16:  QUESTION:  The  internal  interfaces  among  operational  support  organization 
elements  have  been  adequate. 

SPM(OS)  -  017:  QUESTION:  The  procurement  physical  organization  structure  has  been  adequate. 
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SPM(OS)  -  018:  QUESTION:  The  development  contractor  physical  organization  structure  has  been 
adequate. 

SPM(OS)  -  019:  QUESTION:  The  operational  support  physical  organization  structure  has  been 
adequate. 


SOFTWARE  PROJECT  MANAGEMENT  DESIGN  METHODS 


SPM(DM)  ”  001:  QUESTION:  The  procurement  design  analysis  studies  have  provided  adequate 
design  guidelines  for  the  development  contractor. 

SPM(DM)  -  002:  QUESTION:  The  standards  for  software  design  required  by  the  procurement 
activity  are  adequate. 

SPM(DM)  -  003:  QUESTION:  The  software  design  methodology  used  by  the  development  contractor 
is  adequate. 

SPM(DM)  “  004:  QUESTION:  The  design  standards  and  methods  adopted  for  use  by  the  operational 
support  activity  during  postdeployment  software  support  are  adequate. 

SPM(DM)  -  005:  QUESTION:  The  System  Design  Review  process  has  been  adequate. 

SPM(DM)  -  006:  QUESTION:  The  software  requirements  appear  to  be  reasonable. 

SPM(DM)  -  007:  QUESTION:  The  number  of  software  requirements  that  cannot  be  traced  to  an  end 
item  product  is  minimal. 

SPM(DM)  -  008:  QUESTION:  The  number  of  software  requirements  that  cannot  be  tested  is 
minimal. 

SPM(DM)  -  009:  QUESTION:  The  profile  of  changes  to  software  requirements  is  reasonable. 

SPM(DM)  -  010:  QUESTION:  The  profile  of  tmresolved  software  review  action  items  is  reasonable. 

SPM(DM)  -  Oil:  QUESTION:  The  development  contractor  requirements  analysis  process  has  been 
adequate. 

SPM(DM)  “  012:  QUESTION:  The  development  contractor  top-level  design  process  has  been  ade¬ 
quate. 

SPM(DM)  -  013:  QUESTION:  The  development  contractor  detailed  design  process  has  been  ade¬ 
quate. 

SPM(DM)  -  014:  QUESTION:  The  design  completion  of  CSCIs  relative  to  the  software  life  cycle 
development  schedule  has  been  reasonable. 

SPM(DM)  -  015:  QUESTION:  The  development  contractor  monitor  of  the  subcontractor  software 
design  process  has  been  adequate. 
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SPM(DM)  -  016:  QUESTION:  The  design  specifications  for  the  software  products  contain  adequate 
information  to  implement  the  software  with  the  required  functionality  and  within  the  schedule  and 
budget  requirements. 

SPM(DM)  -  017:  QUESTION:  The  operational  support  concept  for  design  of  software  revisions 
during  postdeployment  software  support  is  adequate. 

SPM(DM)  -  018:  QUESTION:  The  operational  support  concept  for  design  review  during  post¬ 
deployment  software  support  is  adequate. 


SOFTWARE  PROJECT  MANAGEMENT  IMPLEMENTATION  METHODS 


SPM(IM)  -  001:  QUESTION:  The  procurement  activity  has  adequately  monitored  the  implementa¬ 
tion  of  the  software  design  specifications. 

SPM(IM)  -  002:  QUESTION:  The  procurement  test  organization  interface  with  the  development 
contractor  is  adequate  enough  to  ensure  success  of  the  system  integration  tests. 

SPM(IM)  -  003:  QUESTION:  The  operational  support  activity  has  been  actively  involved  with  the 
development  contractor's  software  implementation  in  order  to  learn  the  software  prior  to  officially 
accepting  software  support  responsibility. 

SPM(IM)  -  004:  QUESTION:  The  standards  for  software  implementation  required  by  the  procure¬ 
ment  activity  are  adequate. 

SPM(IM)  -  005:  QUESTION:  The  implementation  methodology  used  by  the  development  contractor 
is  adequate. 

SPM(IM)  -  006:  QUESTION:  The  implementation  standards  and  methods  adopted  for  use  by  the 
operational  support  activity  during  postdeployment  software  support  are  adequate. 

SPM(IM)  -  007:  QUESTION:  The  development  contractor  monitor  of  subcontractor  software  im¬ 
plementation  process  has  been  adequate. 

SPM(IM)  -  008:  QUESTION:  The  implementation  completion  of  CSCIs  has  been  reasonable  relative 
to  the  software  life  cycle  schedule. 

SPM(IM)  -  009:  QUESTION:  The  procurement  software  project  management  support  tool  environ¬ 
ment  is  adequate. 

SPM(IM)  -  010:  QUESTION:  The  development  contractor  software  project  management  support  tool 
environment  is  adequate. 

SPM(IM)  -  Oil:  QUESTION:  The  development  contractor  software  configuration  management 
support  tool  environment  is  adequate. 

SPM(IM)  -  012:  QUESTION:  The  development  contractor  system  software  tool  environment  is 
adequate. 

SPM(IM)  -  013:  QUESTION:  The  development  contractor  application  test  software  tool  environ¬ 
ment  is  adequate. 
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SPM(IM)  -  014:  QUESTION:  The  operational  support  software  support  tool  environment  is  ade¬ 
quate. 

SPM(IM)  -  015:  QUESTION:  The  operational  support  software  concept  for  implementation  of 
software  revisions  during  postdeployment  software  support  is  adequate. 

SPM(IM)  -  016:  QUESTION:  The  operational  support  concept  for  implementation  audits  and 
reviews  during  postdeployment  software  support  is  adequate. 

SOFTWARE  PROJECT  MANAGEMENT  TEST  STRATEGIES 


SPM(TS)  -  001:  QUESTION:  The  TEMP  adequately  describes  the  software  test  and  evaluation 
process. 

SPM(TS)  -  002:  QUESTION:  The  software  test  process  for  DT&E  has  followed  the  guidelines  in  the 
TEMP. 

SPM(TS)  -  003:  QUESTION:  The  software  test  process  for  OT&E  has  followed  the  guidelines  in  the 
TEMP. 

SPM(TS)  -  004:  QUESTION:  The  implementation  of  the  software  test  process  by  DT&E  and  OT&E 
organizations  has  been  adequate. 

SPM(TS)  -  005:  QUESTION:  The  test  organizations  have  incorporated  a  strategy  in  their  software 
test  processes  for  coordination  and  sharing  of  test  plans,  procedures,  and  results. 

SPM(TS)  -  006:  QUESTION:  The  requirements  for  the  development  contractor  software  test  strat¬ 
egy  are  clearly  specified  in  the  RFP,  SOW,  and/or  CDRLs. 

SPM(TS)  -  007:  QUESTION:  The  use  of  an  organization  for  software  test  IV&V  support  has  been 
effective. 

SPM(TS)  -  008:  QUESTION:  The  overall  planning  for  software  testing  has  been  adequate. 

SPM(TS)  -  009:  QUESTION:  The  software  test  approach  and  methodologies  employed  are  clearly 
described  in  the  software  test  documentation  and  appear  to  be  effective. 

SPM(TS)  -  010:  QUESTION:  The  software  features  to  be  tested  and  not  to  be  tested  are  clearly 
described  in  the  software  test  documentation. 

SPM(TS)-011:  QUESTION:  The  traceability  software  features  tested/not  tested  to  the  software 
functional  requirements  are  described  in  the  software  test  documentation. 

SPM(TS)  -  012:  QUESTION:  The  software  test  deliverables  are  adequately  specified  in  the  software 
test  documentation. 

SPM(TS)  -  013:  QUESTION:  The  software  test  criteria  used  to  determine  whether  each  test  has 
passed  or  failed  are  clearly  specified  in  the  software  test  documentation. 

SPM(TS)  -  014:  QUESTION:  The  personnel  groups  responsible  for  the  software  tests  are  adequately 
identified  in  the  software  test  documentation. 
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SPM(TS)  -  015:  QUESTION:  The  high  risk  assumptions  of  the  software  testing  approach  along  with 
contingency  plans  for  each  such  assumption  are  adequately  described  in  the  software  test  documen¬ 
tation. 

SPM(TS)  -  016:  QUESTION:  The  schedule  for  software  test  milestones  is  adequately  specified  in  the 
software  test  documentation. 

SPM(TS)  -  017:  QUESTION:  Software  testing  is  adequately  prioritized  in  the  software  test 
approach  according  to  mission  criticality  concerns. 

SPM(TS)  -  018:  QUESTION:  The  software  test  environment  is  adequately  identified  in  the  software 
test  documentation  and  is  adequate  for  accomplishing  the  required  testing. 

SPM(TS)  -  019:  QUESTION:  The  configuration  management  of  the  software  test  process  is  ade¬ 
quate. 

SPM(TS)  -  020:  QUESTION:  The  transition  of  the  software  test  strategy  from  the  development 
contractor  to  the  operational  support  activity  has  been  adequately  addressed  in  the  software  test 
documentation  and  the  procurement  software  test  plans. 


SOFTWARE  PROJECT  MANAGEMENT  PROJECT  INTERFACES 


SPM(PI)-001:  QUESTION: 

SPM(PI)-002:  QUESTION: 

SPM(PI)-003:  QUESTION: 

SPM(PI)-004:  QUESTION: 

SPM(PI)  -  005:  QUESTION: 

SPM(PI)-006:  QUESTION: 

SPM(PI)-007:  QUESTION: 
interfaces  are  adequate. 


The  system  program  office  external  interfaces  are  adequate. 

The  implementing  external  interfaces  are  adequate. 

The  using  command  external  interfaces  are  adequate. 

The  operational  support  activity  external  interfaces  are  adequate. 

The  training  command  external  interfaces  are  adequate. 

The  development  contractor  external  interfaces  are  adequate. 

The  development  test  and  evaluation  (DT&E)  organization  external 


SPM(PI)  -  008:  QUESTION:  The  operational  test  and  evaluation  (OT&E)  organization  external 
interfaces  are  adequate. 


SPM(PI)  -  009:  QUESTION:  The  computer  resources  working  group  (CRWG)  external  interfaces  are 
adequate. 

SPM(PI)  -  010:  QUESTION:  The  test  planning  working  group  (TPWG)  external  interfaces  are 
adequate. 

SPM(PI)  -  Oil:  QUESTION:  The  interface  control  working  group  (ICWG)  external  interfaces  are 
adequate. 

SPM(PI)  -  012:  QUESTION:  The  independent  verification  and  validation  (IV&V)  agency  external 
interfaces  are  adequate. 
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SPM(PI)  -  013:  QUESTION:  The  software  configuration  management  interfaces  among  all  activi¬ 
ties'  management  components  for  the  subject  system  are  adequate. 

SPM(PI)  -  014:  QUESTION:  The  software  quality  assurance  management  interfaces  among  all 
activities'  management  components  for  the  subject  system  are  adequate. 

SPM(PI)  -  015:  QUESTION:  The  contract  management  interfaces  among  all  activities'  management 
components  for  the  subject  system  are  adequate. 

SPM(PI)  -  016:  QUESTION:  The  interservice  external  interfaces  with  all  activities'  management 
components  are  adequate. 

SOFTWARE  CONFIGURATION  MANAGEMENT  IDENTIFICATION 


SCM(ID)  -  001:  QUESTION:  The  procurement  policy,  standards,  and  conventions  applied  to  the 
identification  of  software  configuration  items  are  adequate. 

SCM(ID)  -  002:  QUESTION:  The  procurement  identification  of  deliverable  software  configuration 
items  is  adequate. 

SCM(ID)  -  003:  QUESTION:  The  procurement  activity  identification  of  the  software  configuration 
baselines  is  adequate. 

SCM(ID)  -  004:  QUESTION:  The  system/segment  specification  adequately  identifies  elements  of  the 
software  functional  baseline. 

SCM(ID)  -  005:  QUESTION:  The  performance  requirement  specifications  adequately  identify 
elements  of  the  software  allocated  baseline. 

SCM(ID)  -  006:  QUESTION:  The  implementation  specifications  adequately  identify  elements  of  the 
software  product  baseline. 

SCM(ID)-007:  QUESTION:  The  identifier  characteristics  for  software  configuration  item  names 
are  adequate. 

SCM(ID)  -  008:  QUESTION:  The  development  contractor  internal  identifier  naming  standards/ 
conventions  satisfy  contractual  regulations. 

SCM(ID)  -  009:  QUESTION:  Development  contractor  identification  standards  and  conventions  can 
be  transitioned  to  operational  support  standards  and  conventions. 

SCM(ID)  -  010:  QUESTION:  Development  contractor  deliverable  configuration  items  are  named  to 
adequately  identify  multiple  versions  and  variations. 

SCM(ID)  -  Oil:  QUESTION:  Development  contractor  identification  procedures  are  structured  to 
permit  easy  addition,  deletion,  or  modification  of  configured  items  at  any  hierarchical  level. 

SCM(ID)  -  012:  QUESTION:  Development  contractor  identification  procedures  for  addition,  dele¬ 
tion,  and  modification  of  configured  items  are  being  followed. 
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SCM(ID)  -  013:  QUESTION:  The  physical  medium  of  configured  items  is  adequately  described  by 
the  development  contractor  software  component/item  identification  scheme. 

SCM(ID)  -  014:  QUESTION:  The  development  contractor  software  identifiers  adequately  distin¬ 
guish  among  different  states  (e.g.,  course,  object,  load,  core  images,  listings)  of  the  software. 

SCM(ID)  -  015:  QUESTION:  The  development  contractor  software  change  control  forms  are  ade¬ 
quate. 

SCM(ID)  -  016:  QUESTION:  Subcontractor  configuration  item  identification  practices  are  moni¬ 
tored  by  the  development  contractor. 

SCM(ID)  -  017:  QUESTION:  The  documentation  which  collectively  identifies  the  content  of  a  con¬ 
figuration  item  is  adequate. 

SCM(ID)-018:  QUESTION:  Software  configured  items  that  implement  safety  provisions  are 
adequately  identified. 

SCM(ID)  -  019:  QUESTION:  Software  configured  items  that  implement  computer/communication 
security  provisions  are  adequately  identified. 

SCM(ID)  -  020:  QUESTION:  The  identification  requirements  for  postdeployment  support  are  ade¬ 
quately  addressed  in  the  CRLCMP. 

SCM(ID)  -  021:  QUESTION:  The  automated  support  tools  for  postdeployment  support  of  configura¬ 
tion  identification  are  adequately  addressed  in  the  CRLCMP . 


SOFTWARE  CONFIGURATION  MANAGEMENT  CONTROL 


SCM(CC)  -  001:  QUESTION:  The  procurement  policy,  standards,  and  conventions  applied  to  the 
control  of  software  configuration  items  are  adequate. 

SCM(CC)  -  002:  QUESTION:  The  procurement  activity  has  implemented  adequate  software  con¬ 
figuration  management,  based  upon  regulations,  to  control  the  functional  and  physical  characteris¬ 
tics  of  all  CSCIs. 

SCM(CC)-003:  QUESTION:  The  procurement  configuration  management  planning  documents 
contain  sufficient  guidance  for  configuration  control. 

SCM(CC)  -  004:  QUESTION:  The  development  contractor  configuration  management  activities  are 
adequately  monitored  by  the  procurement  activity. 

SCM(CC)  -  005:  QUESTION:  The  procurement  configuration  control  procedures  for  the  Class  I  and 
Class  II  changes  (or  equivalent  categories)  are  adequate. 

SCM(CC)  -  006:  QUESTION:  The  use  of  deviations  and  waivers  by  the  development  contractor 
which  could  affect  the  supportability  of  the  software  has  been  adequately  controlled  by  the  procure¬ 
ment  activity. 

SCM(CC)  -  007:  QUESTION:  The  procurement  baseline  control  forms  are  adequate. 
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SCM(CC)  -  008:  QUESTION:  The  procurement  configuration  control  board  procedures  are  ade¬ 
quate, 

SCM(CC)  -  009:  QUESTION:  The  procurement  procedures  for  turnover  and  transfer  of  configura¬ 
tion  control  to  the  operational  support  activity  have  been  adequately  planned. 

SCM(CC)  -  010:  QUESTION:  Development  contractor  configuration  control  standards  and  conven¬ 
tions  can  be  transitioned  to  operational  support  standards  and  conventions. 

SCM(CC)  -  Oil:  QUESTION:  The  development  contractor  configuration  control  board  has  an  ade¬ 
quate  interface  with  the  procurement  activity  configuration  control  board. 

SCM(CC)  -  012:  QUESTION:  The  development  contractor  configuration  control  board  procedures 
are  adequate  to  distinguish  between  hardware  and  software  failures. 

SCM(CC)  -  013:  QUESTION:  The  development  contractor  configuration  control  procedures  can  be 
transitioned  to  or  are  compatible  with  the  operational  support  activity  planned  configuration  control 
procedures. 

SCM(CC)  -  014:  QUESTION:  The  development  contractor  automated  support  tools  for  configuration 
control  of  baselines  and  internal  development  identifications  is  adequate. 

SCM(CC)  -  015:  QUESTION:  The  development  contractor  software  change  control  forms  are  ade¬ 
quate. 

SCM(CC)  -  016:  QUESTION:  Subcontractor  configuration  item  control  practices  are  monitored  by 
the  development  contractor. 

SCM(CC)  -  017:  QUESTION:  Configured  items  that  implement  safety  provisions  are  adequately 
controlled. 

SCM(CC)  -  018:  QUESTION:  Software  configured  items  that  implement  computer/communications 
security  provisions  are  adequately  controlled. 

SCM(CC)  -  019:  QUESTION:  Distribution  of  configured  item  changes  from  the  operational  support 
activity  to  the  field  is  adequately  controlled. 

SCM(CC)  -  020:  QUESTION:  The  configuration  control  responsibility  for  integrating  computer 
resources  into  the  system  has  remained  centralized  throughout  the  life  of  the  system. 

SCM(CC)  -  021:  QUESTION:  The  configuration  control  requirements  for  postdeployment  support 
are  adequately  addressed  in  the  CRLCMP. 

SCM(CC)  -  022:  QUESTION:  The  operational  support  configuration  control  boards  are  adequately 
defined  to  handle  software  changes. 

SCM(CC)  -  023:  QUESTION:  The  automated  support  tools  for  postdeployment  support  of  configura¬ 
tion  control  are  adequately  addressed  in  the  CRLCMP. 
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SOFTWARE  CONFIGURATION  MANAGEMENT  STATUS  ACCOUNTING 


SCM(SA)  -  001:  QUESTION:  The  procurement  policy,  standards,  and  conventions  applied  to  the 
configuration  status  accounting  of  software  configuration  items  are  adequate, 

SCM(SA)  -  002:  QUESTION:  The  procurement  activity  has  implemented  adequate  software  con¬ 
figuration  status  accounting,  based  on  regulations,  to  report  the  functional  and  physical  characteris¬ 
tics  of  all  CSCIs. 

SCM(SA)  -  003:  QUESTION:  The  procurement  configuration  management  planning  documents 
contain  sufficient  guidance  for  configuration  status  accounting, 

SCM(SA)  -  004:  QUESTION:  The  procurement  activity  configuration  status  accounting  procedures 
are  adequate, 

SCM(SA)  -  005:  QUESTION:  The  development  contractor  internal  configuration  status  accounting 
procedures  are  adequate. 

SCM(SA)  -  006:  QUESTION:  The  development  contractor  configuration  status  accounting  has  an 
adequate  interface  with  the  procurement  activity  configuration  status  accounting, 

SCM(SA)-007:  QUESTION:  The  development  contractor  configuration  status  accounting  proce¬ 
dures  can  be  transitioned  to  or  are  compatible  with  the  operational  support  activity  planned  configu¬ 
ration  status  accounting  procedures. 

SCM(SA)  -  008:  QUESTION:  The  development  contractor  automated  support  tools  for  configuration 
status  accounting  of  baselines  and  internal  development  identifications  are  adequate. 

SCM(SA)  -  009:  QUESTION:  The  development  contractor  software  configuration  status  accounting 
forms  are  adequate. 

SCM(SA)  -  010:  QUESTION:  Subcontractor  configuration  item  configuration  status  accounting 
procedures  are  monitored  by  the  development  contractor. 

SCM(SA)  -  oil:  QUESTION:  Status  of  software  configuration  items  that  implement  safety  provi¬ 
sions  is  adequately  reported. 

SCM(SA)  -  012:  QUESTION:  Status  of  software  configured  items  that  implement  com¬ 
puter/communications  security  provisions  is  adequately  reported, 

SCM(SA)  -  013:  QUESTION:  The  configuration  status  accounting  requirements  for  postdeployment 
support  are  adequately  addressed  in  the  CRLCMP. 

SCM(SA)  -  014:  QUESTION:  The  operational  support  configuration  status  accounting  procedures 
are  adequately  defined  to  handle  software  change  reporting  requirements. 

SCM(SA)  -  015:  QUESTION:  The  automated  support  tools  for  postdeployment  support  of  configur¬ 
ation  status  accounting  are  adequately  addressed  in  the  CRLCMP. 
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SOFTWARE  CONFIGURATION  MANAGEMENT  AUDIT/REVIEW 


SCM(AR)  -  001:  QUESTION:  The  procurement  policy,  standards,  and  conventions  applied  to  the 
audit  and  review  of  software  configuration  items  are  adequate. 

SCM(AR)  -  002:  QUESTION:  The  procurement  activity  has  implemented  adequate  software 
configuration  audit/review  based  on  regulations  to  control  the  functional  and  physical  characteristics 
ofallCSCIs. 

SCM(AR)  -  003:  QUESTION:  The  procurement  configuration  management  planning  documents 
contain  sufficient  guidance  for  configuration  audit/review. 

SCM(AR)  -  004:  QUESTION:  The  conduct  of  formal  reviews  and  audits  follows  a  format  based  on 
the  checklists  from  MIL-STD-1521B,  appropriately  tailored  for  the  specific  software  audit/review. 

SCM(AR)  -  005:  QUESTION:  The  software  product  acceptance  requirements  are  adequate. 

SCM(AR)  -  006:  QUESTION:  The  development  contractor  internal  configuration  audit/review  proc¬ 
ess  facilitates  the  development  of  high  quality  production  software. 

SCM(AR)  -  007:  QUESTION:  Configuration  audit/review  interfaces  among  procurement,  develop¬ 
ment  contractor,  and  operational  support  activities  are  adequate. 

SCM(AR)  -  008:  QUESTION:  The  development  contractor  configuration  management  tool  support 
facilitates  the  audit/review  of  the  process  by  which  changes  are  incorporated  into  configuration  iden¬ 
tifications. 

SCM(AR)  -  009:  QUESTION:  Subcontractor  configuration  item  audit/review  practices  are  monitored 
by  the  development  contractor. 

SCM(AR)  -  010:  QUESTION:  Configured  items  that  implement  safety  provisions  are  adequately 
audited  and  reviewed. 

SCM(AR)  -  Oil:  QUESTION:  Software  configured  items  that  implement  computer/communications 
security  provisions  are  adequately  audited  and  reviewed. 

SCM(AR)  -  012:  QUESTION:  The  software  configuration  audit/review  requirements  for  post¬ 
deployment  support  are  adequately  addressed  in  the  CRLCMP. 

SCM(AR)  -  013:  QUESTION:  The  automated  support  tools  for  postdeployment  support  of  configu¬ 
ration  audit/review  are  adequately  addressed  in  the  CRLCMP. 
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QUESTION  NUMBER  LIST  BY  TIME  PHASE 


This  attachment  contains  a  list  of  the  Software  Support  Life  Cycle  Process  question  numbers  that 
should  be  answerable  at  the  various  time  phases  or  milestone  reviews. 


The  reviews  are: 

SRR 

System  Requirements  Review 

SDR 

System  Design  Review 

SSR 

Software  Specification  Review 

PDR 

Preliminary  Design  Review 

CDR 

Critical  Design  Review 

TRR  (DT&E) 

Test  Readiness  Review  for  Development  Test  and  Evaluation  (About  the 
same  timeframe  as  the  Functional  Configuration  Audit  (FCA) 

TRR  (OT&E) 

Test  Readiness  Review  for  Operational  Test  and  Evaluation  (Normally 
after  the  Physical  Configuration  Audit  (PCA)  and  the  Formal  Qualifica¬ 
tion  Review/Test  (FQR/T)) 

OT&E 

Operational  Test  and  Evaluation 

The  activities  to  be  emphasized  at  the  appropriate  review  are: 

P  Procurement  activities  (events/actions/documents  that  are  procurement 

in  nature) 

DC  Development  Contractor  activities  (events/actions/documents  that  are  the 

responsibility  of  the  developer) 

OS  Operational  Support  activities  (events/actions/documents  that  impact  the 

postdeplo5mient  software  support  of  a  system) 
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TRR 

TRR 

Question  Number 

SRR  SDR  SSR 

PDR 

CDR 

DT&E 

OT&E 

OT&E 

SPM(PL)-001 

P 

OS 

SPM(PL)-002 

P 

OS 

SPM(PL)-003 

P 

OS 

SPM(PL)-004 

P 

DC 

OC 

SPM(PL)-005 

P 

SPM(PL)-006 

P 

SPM(PL)-007 

P 

DC 

OS 

SPM(PL)-008 

P 

DC 

OS 

SPM(PL)-009 

P 

OS 

SPM(PL)-010 

P 

OS 

SPM(PL)-011 

P 

DC 

SPM(PL)-012 

P 

OS 

SPM(PL)-013 

P 

OS 

SPM(PL)-014 

P 

OS 

SPM(PL)-015 

P 

DC 

OS 

SPM(PL)-016 

P 

SPM(PL)-017 

P 

SPM(PL)-018 

P 

DC 

OS 

SPM(PL)-019 

OS 

SPM(PL)-020 

P 

OS 

SPM(PL)-021 

P 

OS 

SPM(PL)-022 

P 

OS 

SPM(PL)-023 

P 

OS 

SPM(PL)-024 

P 

OS 

SPM(PL)-025 

P 

OS 

SPM(PL)-026 

P 

DC 

OS 

SPM(PL)-027 

P 

DC 

OS 

SPM(PL)-028 

P 

DC 

OS 

SPM(PL)-029 

P 

DC 

OS 

SPM(PL)-030 

DC 

SPM(PL)-031 

DC 

SPM(PL)-032 

DC 

SPM(OS)-001 

P 

DC 

SPM(OS)-002 

P 

DC 

OS 

SPM(OS)-003 

P 

DC 

OS 

SPM(OS)-004 

ALL 

SPM(OS)-005 

P 

SPM(OS)-006 

P 

SPM(OS)-007 

P 

SPM(OS)-008 

DC 

SPM(OS)-009 

DC 

SPM(OS)-010 

DC 

SPM(OS)-Oll 

OS 

SPM(OS)-O12 

OS 

SPM(OS)-O13 

OS 

SPM(OS)-O14 

P 

P 

SPM(OS)-O15 

DC 

DC 

SPM(OS)-O16 

OS 

OS 

SPM(OS)-O17 

P 

P 

SPM(OS)-O18 

DC 

DC 
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SPM(OS)-O19  OS 

TRR  TRR 

Question  Number  SRR  SDR  SSR  PDR  CDR  DT&E  OT&E 


SPM(DM)-001 

SPM(DM)-002 

SPM(DM)-003 

SPM(DM)-004 

SPM(DM)-005 

SPM(DM)-006 

SPM(DM)-007 

SPM(DM)-008 

SPM(DM)-009 

SPM(DM)-010 

SPM(DM)-011 

SPM(DM)-012 

SPM(DM)-013 

SPM(DM)-014 

SPM(DM)-015 

SPM(DM)-016 

SPM(DM)-017 

SPM(DM)-018 


P 

P 

DC 


P-OS 

P-OS 


ALL 


ALL 

ALL 

P 


DC 


DC 


DC 


ALL  ALL 

OS 
OS 


OS 


P-OS 


ALL 

DC 


DC 


DC 


SPM(IM)-001 

SPM(IM)-002 

SPM(IM)-003 

SPM(IM)-004  P 

SPM(IM)-005 

SPM(IM)-006 

SPM(IM)-007 

SPM(IM)-008 

SPM(IM)-009  P 

SPM(IM)-010 

SPM(IM)-011 

SPM(IM)-012 

SPM(IM)-013 

SPM(IM)-014 

SPM(IM)-015 

SPM(IM)-016 


P 

P-DC 

OS 


DC 


OS 

DC 

DC 


DC 

DC 

DC 


OS 


OS 


SPM(TS)-001 

SPM(TS)-002 

SPM(TS)-003 

SPM(TS)-004 

SPM(TS)-005 

SPM(TS)-006 

SPM(TS)-007 

SPM(TS)-008 

SPM(TS)-009 

SPM(TS)-010 

SPM(TS)-011 

SPM(TS)-012 


P 


P-OS 


p 

p 

p 

p 


OS 

P-OS 

p 

ALL 

P-OS 

DC 

DC 

DC 

DC 

ALL 


OS 

OT&E 


P-OS 


OS 

OS 


OS 

OS 


P-OS 

OS 


OS 

OS 

OS 

OS 
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SPM(TS)-013 

SPM(TS)-014 

SPM(TS)-015 


Question  Number 


SPM(TS)-016 

SPM(TS)-017 

SPM(TS)-018 

SPM(TS)-019 

SPM(TS)-020 


ALL 

P-DC 

OS 

P-DC 

TRR 

TRR 

OS 

SRR 

SDR 

SSR 

PDR 

CDR 

DT&E 

OT&E 

OT&E 

P-DC 

P-DC 

P-DC 

OS 

P-DC 

OS 

P-DC 

OS 

P 

DC-OS 

SPM(PI)-001 

SPM(PI)-002 

SPM(PI)-003 

SPM(PI)-004 

SPM(PI)-005 

SPM(PI)-006 

SPM(PI)-007 

SPM(PI)-008 

SPM(PI)-009 

SPM(PI)-010 

SPM(PI)-011 

SPM(PI)-012 

SPM(PI)-013 

SPM(PI)-014 


P 

P 


P-OS 

DC 

P 

P 

ALL 

ALL 


P-DC 

P-DC 


OS 

OS 


ALL 

ALL 

ALL 


SCM(ID)-001  P 

SCM(ID)-002  P 

SCM(ID)-003 

SCM(ID)-004  P 

SCM(ID)-005 

SCM(ID)-006 

SCM(ID)-007 

SCM(ID)-008 

SCM(ID)-009 

SCM(ID)-010 

SCM(ID)-011 

SCM(ID)-012 

SCM(ID)-013 

SCM(ID)-014 

SCM(ID)-015 

SCM(ID)-016 

SCM(ID)-017 

SCM(ID)-018 

SCM(ID)-019 

SCM(ID)-020 

SCM(ID)-021 


P 

P 

DC 

DC 

DC 

DC 

DC 

DC 

DC 

DC 

DC 

DC 

DC 

DC 

DC 

DC 

OS 

OS 


SCM(CC)-001  P 

SCM(CC)-002  P 

SCM(CC)-003  P 

SCM(CC)-004 

SCM(CC)-005 


P 

P 
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SCM(CC)-006 

SCM(CC)-007 

SCM(CC)-008 

SCM(CC)-009 

P 

P 

P 

P 

TRR 

TRR 

Question  Number 

SRR 

SDR 

SSR 

PDR 

CDR 

DT&E 

OT&E  OT&E 

SCM(CC)-010 

SCM(CC)-011 

SCM(CC)-012 

SCM(CC)-013 

SCM(CC)-014 

SCM(CC)-015 

SCM(CC)-016 

SCM(CC)-017 

SCM(CC)-018 

SCM(CC)-019 

SCM(CC)-020 

SCM(CC)-021 

SCM(CC)-022 

SCM(CC)-023 


OS 

OS 


DC 

DC 

DC 


OS 


DC 

DC 

DC 


DC 

DC 

OS 

ALL 

OS 

OS 


SCM(SA)-001  P 

SCM(SA)-002 

SCM(SA)-003  P 

SCM(SA)-004 

SCM(SA)-005 

SCM(SA)-006 

SCM(SA)-007 

SCM(SA)-008 

SCM(SA)-009 

SCM(SA)-010 

SCM(SA)-011 

SCM(SA)-012 

SCM(SA)-013 

SCM(SA)-014 

SCM(SA)-015 


P 


DC 

DC 


DC 

DC 

DC 


OS 


DC 


DC 

DC 

OS 

OS 


SCM(AR)-001 

SCM(AR)-002 

SCM(AR)-003 

SCM(AR)-004 

SCM(AR)-005 

SCM(AR)-006 

SCM(AR)-007 

SCM(AR)-008 

SCM(AR)-009 

SCM(AR)-010 

SCM(AR)-011 

SCM(AR)-012 

SCM(AK)-013 


P 

P 


P 


P 


OS 


p 


DC 


DC 


DC 

DC 

DC 

DC 


OS 
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GLOSSARY 


The  glossary  of  terms  for  software  supportability  life  cycle  processes  are  based  on  a  number  of 
sources.  Some  terms  have  more  than  one  description.  When  this  is  the  case,  the  descriptions  either: 

a.  Are  significantly  different  between  sources  (though  the  effective  meaning  may  not  be 
much  different). 

b.  Are  used  differently  (different  users  or  technical  language). 

c.  May  be  found  within  the  context  of  a  different  source. 

d.  Have  real  differences  in  meaning. 

The  source  of  each  description  is  indicated  by  a  symbol  in  parentheses  before  that  source's  term 
description.  Examples  of  the  symbols  used  and  corresponding  sources  are: 

(AFOTECP  1)  AFOTECP  800-2,  Volume  1,  Management  of  Software  Operational  Test  and 
Evaluation. 

(AFOTECP  3)  AFOTECP  800-2,  Volume  3,  Software  Maintainability  Evaluator's  Guide. 
(AFOTECP  5)  AFOTECP  800-2,  Volume  5,  Software  Support  Resources  Evaluation  Guide. 

(APR  55-43)  Air  Force  Regulation  55-43,  Management  of  Operational  Test  And  Evaluation. 

(AFR  80-14)  Air  Force  Regulation  80-14,  Test  and  Evaluation. 

(AFR  800-14)  Air  Force  Regulation  800-14,  Lifecycle  Management  of  Computer  Resources  in 
Systems. 

(DoD  80A)  Department  of  Defense  Standard  480A,  Configuration  Control  -  Engineering 
Changes,  Deviations,  and  Waivers. 

(DoDD  5000.2-M)  Department  of  Defense  Directive  5000.2-M,  Defense  Acquisition  Management 
Documentation  and  Reports. 

(ROWE)  Rowe,  William,  An  Anatomy  of  Risk.  John  Wiley,  1977. 

(CURRENT)  Current  document  definition. 
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TERMS 


Allocated  Baseline 

(DoD  480A) 

See  Baseline. 

Allocated  Configuration  Identification 

(DoD  480A) 

Current,  approved  performance  oriented  specifications  governing  the  development  of 
configuration  items  that  are  part  of  a  higher  level  configuration  item  (Cl),  in  which  each  specification 
(1)  defines  the  functional  characteristics  that  are  allocated  from  those  of  the  higher  level  Cl,  (2) 
establishes  the  tests  required  to  demonstrate  achievement  of  its  allocated  functional  characteristics, 
(3)  delineates  necessary  interface  requirements  with  other  associated  configuration  items,  and  (4) 
establishes  design  constraints,  if  any,  such  as  component  standardization,  use  of  inventory  items, 
and  integrated  logistic  support  requirements. 

Application  Software 

(AFOTECP  5) 

The  software  written  by  support  personnel,  or  purchased  from  a  contractor,  used  directly  in 
supporting  mission  critical  computer  resources  (MCCR).  It  is  normally  used  for  simulation,  testing, 
and  MCCR  code  development. 

Automated  Software  Development  Tool 

(AFOTECP  5) 

A  component  of  the  System  Software  that  assists  in  the  design,  implementation,  documentation, 
and  verification  of  MCCR  software. 

Availability 

(CURRENT) 

A  measure  of  the  degree  to  which  an  item  is  in  the  operable  and  committable  state  at  the  start  of 
the  mission  when  the  mission  is  called  for  at  an  unknown  (random)  time.  (MIL-STD-721) 

(AFOTECP  5) 

The  probability  that  a  system  is  operating  satisfactorily  at  any  point  in  time  when  used  imder 
stated  conditions. 

Baseline 

(DoD  480A) 

A  configuration  identification  docmnent  or  a  set  of  such  documents  formally  designated  and 
fixed  at  a  specific  time  during  a  Cl's  life  cycle.  Baselines,  plus  approved  changes  from  those 
baselines,  constitute  the  current  configuration  identification.  For  configuration  management,  there 
are  three  baselines,  as  follows: 

(1)  Functional  Baseline.  The  initial  approved  functional  configuration  identification. 

(2)  Allocated  Baseline.  The  initial  approved  allocated  configuration  identification. 
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(3)  Product  Baseline.  The  initial  approved  or  conditionally  approved  product  configura¬ 
tion  identification. 

(ROWE) 

A  known  reference  used  as  a  guide  for  further  development  activities. 

Baseline  Profile 

(CURRENT) 

See  Baseline  Software  Change  Profile. 

Baseline  Software  Change  Profile 

(CURRENT) 

The  set  of  numbers  (or  any  subset)  determined  by  specifying  the  number  of  change  requests  per 
release  for  each  change  request  category.  A  change  request  category  is  the  triple  (maintenance  type, 
maintenance  priority,  maintenance  complexity)  where  maintenance  type  is  correction,  enhancement, 
or  conversion;  maintenance  priority  is  normal,  urgent,  or  emergency;  and  maintenance  complexity  is 
low,  medium,  or  high. 

Baseline  Software  Supportability  Estimate 

(CURRENT) 

See  User/Supporter  Baseline  Estimate 

Block  Release 

(CURRENT) 

See  Release. 

Change  Control 

(DoD  480A) 

See  Configuration  Control. 

Computer  Program 
(AFR  800-14) 

A  series  of  instructions  or  statements  in  a  form  acceptable  to  an  electronic  computer,  designed  to 
cause  the  computer  to  execute  an  operation  or  operations. 

Computer  Program  Configuration  Item  (CPCI) 

(CURRENT) 

See  Configuration  Item.  CPCI  has  been  replaced  by  the  term  CSCI  (below). 

Computer  Resources 
(AFR  800-14) 

The  totality  of  computer  equipment,  computer  programs,  associated  documentation,  contractual 
services,  personnel,  and  supplies. 
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Computer  Resources  Lifecycle  Management  Plan 
(AFR  800-14) 

The  CRLCMP  identifies  computer  resources  development  strategy,  documents  the  software 
support  concept  and  the  resources  needed  to  achieve  that  support  posture  (organizational 
relationships  and  responsibilities  for  management  and  technical  support),  identifies  the  applicable 
directives  (regulations,  operating  instructions,  technical  orders,  etc.),  and  defines  any  changes  or  new 
directives  needed  for  operation  or  support  of  computer  resources  in  a  system. 

Computer  Resources  Working  Group  (CRWG) 

(CURRENT) 

A  group  comprised  of  all  the  participating  commands  (for  a  particular  system)  which  writes  and 
updates  the  CRLCMP.  The  group  ensures  that  necessary  elements  of  the  CRLCMP  are  included  in 
the  transfer  or  turnover  agreements. 

Computer  Software  Configuration  Item  (CSCI) 

(CURRENT) 

See  Configuration  Item. 

Configuration  Audit 

(CURRENT) 

The  process  of  verifying  that  all  required  configuration  items  have  been  produced,  that  the 
current  version  agrees  with  specified  requirements,  that  the  technical  documentation  completely  and 
accurately  describes  the  configuration  items,  and  that  all  change  requests  have  been  resolved. 

Configuration  Control 

(DoD  480A) 

The  systematic  evaluation,  coordination,  approval  or  disapproval,  and  implementation  of  all 
approved  changes  in  the  configuration  of  a  configuration  item  after  formal  establishment  of  its 
configuration  identification. 

Configuration  Identification 

(DoD  480A) 

The  current  approved  or  conditionally  approved  technical  documentation  for  a  configuration 
item  as  set  forth  in  specifications,  drawings  and  associated  lists,  and  documents  referenced  therein. 

Configuration  Index 

(CURRENT) 

The  document  produced  by  the  development  contractor  that  reports  the  current  status  of 
configuration  item  development  in  terms  of  specifications  and  other  documents  that  depend  on  the 
configuration,  such  as  qualification  Test  Plans  and  Procedures,  User  Manuals,  and  the  Version 
Description  Document.  It  lists  all  Engineering  Change  Proposals  (ECP)  and  Software  Change 
Notices  (SCN)  incorporated,  approved  ECPs  not  yet  incorporated,  and  other  data. 

Configuration  Item  (Cl) 


(AFR  800-14) 
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An  aggregation  of  equipment/software,  or  any  of  its  discrete  portions,  which  satisfies  an  end  use 
function  and  is  designated  by  the  government  for  configuration  management.  CIs  may  vary  widely 
in  complexity,  size,  and  type  from  an  aircraft  or  electronic  system  to  a  test  meter  of  round  of 
ammunition.  During  development  and  initial  production,  CIs  are  only  those  specification  items  that 
are  referenced  directly  in  the  contract  (or  an  equivalent  in-house  agreement).  During  the  operation 
and  maintenance  period,  any  repairable  item  designated  for  separate  procurement  is  a  configuration 
item  (AFR  65-3). 

Configuration  Management  (CM) 

(DoD  480A) 

A  discipline  applying  technical  and  administrative  direction  and  surveillance  to  (1)  identify  and 
document  the  functional  and  physical  characteristics  of  a  configuration  item,  (2)  control  changes  to 
those  characteristics,  and  (3)  record  and  report  change  processing  and  implementation  status. 

Configuration  Management  Plan  (CMP) 

(CURRENT) 

A  document  that  describes  project  responsibilities  and  procedures  for  implementing  CM. 

Configuration  Management  System  (CMS) 

(AFOTECP  5) 

A  system  applying  technical  and  administrative  direction  and  surveillance  to  identify  and 
document  the  functional  and  physical  characteristics  of  a  configuration  item,  to  control  changes  to 
those  characteristics,  and  to  record  and  report  change  processing  and  implementation  status. 

Configuration  Status  Accounting 

(DoD  480A) 

The  recording  and  reporting  of  the  information  that  is  needed  to  manage  a  configuration 
effectively,  including  a  listing  of  the  approved  configuration  identification,  the  status  of  proposed 
changes  to  the  configuration,  and  the  implementation  status  of  approved  changes. 

Consistency 

(CURRENT) 

A  measure  of  the  extent  the  software  products  correlate  and  contain  uniform  notation, 
terminology,  and  symbology. 

Contract  Logistics  Support  (CLS) 

(AFR  800-21) 

CLS  is  a  method  used  to  provide  all  or  part  of  a  system's  logistics  support  by  contract 
throughout  its  entire  lifecycle.  CLS  is  a  support  concept  rather  than  an  acquisition  technique. 

Critical  Issues 

(DoDD  5000.2) 

Those  questions  relating  to  a  system's  operational,  technical,  support,  or  other  capability,  that 
must  be  answered  before  the  system's  overall  worth  can  be  estimated/evaluated,  and  that  are  of 
primary  importance  to  the  decision  authority  in  allowing  the  system  to  advance  into  the  next  acquisi¬ 
tion  phase. 
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Data  Item  Description 

(AFR  800-14) 

A  form  that  specifies  an  item  of  data  required  to  be  furnished  by  a  contractor.  This  form 
specifically  defines  the  content,  preparation  instructions,  format,  and  intended  use  of  each  data 
product. 

Descriptiveness 

(CURRENT) 

A  measure  of  the  extent  that  software  products  contain  information  regarding  its  objectives, 
assumptions,  inputs,  processing,  outputs,  components,  revision  status,  etc. 


Development  Contractor  Activity 

(CURRENT)  .  .  . 

Those  organizations  responsible  for  development  of  a  system  in  order  to  achieve  an  initial 
operational  capability.  Organizations  include  the  prime  development  contractor  and  any 
subcontractors  to  the  prime  contractor. 

Documentation 

(AFOTECP  5) 

All  of  the  written  work  describing  operating  and  maintenance  procedures  for  a  system. 
Engineering  Change  Proposal  (ECP) 

(AFR  55-43) 

A  formal,  priced  document  (DD  Form  1692)  used  to  propose  changes  to  the  contract  provisions 
and  scope,  if  not  partially  waived  (see  Contract  Change  Proposal),  and  to  the  configuration  item 
baseline  identification  especially  when  related  equipment,  critical  issues,  interfaces,  or  technical 
manuals  are  affected  or  retrofit  is  involved.  See  MIL-STDs  480,  481,  and  483  and  AFR  400-3. 

Evaluation 

(AFR  80-14)  ^  . 

The  review  and  analysis  of  qualitative  or  quantitative  data  obtained  from  design  review, 

hardware  inspection,  testing,  or  operational  usage  of  equipment. 

(ROWE)  .  .  j  .  *  r 

Comparison  of  an  activity  performance  with  the  objectives  of  the  activity  and  assignment  of  a 

success  measure  to  that  performance. 

Evaluation  Criteria 

(DoDD  5000.2)  .  ,  u  v* 

Standards  by  which  achievement  of  required  technical  and  operational  effectiveness/suitability 

characteristics,  or  resolution  of  technical  or  operational  issues,  may  be  evaluated.  At  Milestones  I 
and  II,  evaluation  criteria  should  include  quantitative  thresholds  for  the  IOC  system.  At  Milestone 
III  and  beyond  (or  IOC,  whichever  occurs  first),  evaluation  criteria  should  include  quantitative 
thresholds  for  the  mature  system.  If  system  maturity  is  greater  than  2  years  beyond  IOC, 
intermediate  evaluation  criteria,  appropriately  time-lined,  must  also  be  provided. 
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Expandability 

(CURRENT) 

A  measure  of  the  extent  that  a  physical  change  to  information,  computational  functions,  data 
storage,  or  execution  time  can  be  easily  accomplished  once  the  nature  of  what  is  to  be  changed  is 
understood. 

(AFOTECP  5) 

A  measure  of  the  ease  with  which  the  functional  capability  of  computer  hardware  or  software 
may  be  expanded. 

Facility 

(AFOTECP  5) 

The  physical  plant  and  the  services  it  provides;  specific  examples  are  physical  space,  electrical 
power,  physical  and  electromagnetic  (TEMPEST)  security,  environmental  control,  fire  safety 
provisions,  and  communications  availability. 

Firmware 

(AFOTECP  1) 

(1)  Computer  programs  and  data  loaded  in  a  class  of  memory  that  cannot  be  dynamically 
modified  by  the  computer  during  processing. 

(2)  Hardware  that  contains  a  computer  program  and  data  that  cannot  be  changed  in  its 
application  environment. 

Note:  Computer  programs  and  data  contained  in  firmware  are  classified  as  software;  the 
circuitry  containing  the  computer  program  and  data  is  classified  as  hardware  (Data  and  Analysis 
Center  for  Software). 

Functional  Baseline 

(DoD  480A) 

See  Baseline 

Functional  Configuration  Audit  (FCA) 

(DoD  480A) 

The  formal  examination  of  functional  characteristics  test  data  for  a  configuration  item,  prior  to 
acceptance,  to  verify  that  the  item  has  achieved  the  performance  specified  in  its  functional  or 
allocated  configuration  identification. 

Functional  Configuration  Identification 

(DoD  480A) 

The  current  approved  technical  documentation  for  a  configuration  item  that  prescribes  (1)  all 
necessary  functional  characteristics,  (2)  the  tests  required  to  demonstrate  achievement  of  specified 
functional  characteristics,  (3)  the  necessary  interface  characteristics  with  associated  CIs,  (4)  the  CFs 
key  functional  characteristics  and  its  lower  level  CIs,  if  any,  and  (5)  design  constraints,  such  as 
envelope  dimensions,  component  standardization,  use  of  inventory  items,  and  integrated  logistics 
support  policies. 
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Historical  Maintenance  Profile 

(CURRENT) 

A  histogram  of  data  on  software  system  releases,  with  the  x-axis  representing  discrete  ranges  of 
(available)  person-months  per  change  and  the  y-axis  representing  the  number  of  software  system 
releases  that  fall  into  each  x-axis  discrete  range.  For  purposes  of  analysis  or  illustration,  the  axes 
may  be  reversed. 

Independent  Verification  and  Validation  (IV&V) 

(AFOTECP  1) 

An  independent  assessment  process  structured  to  ensure  that  computer  programs  fulfill  the 
requirements  stated  in  system  and  subsystem  specification  and  satisfactorily  perform  the  functions 
required  to  meet  the  user's  and  supporter's  requirements.  IV&V  consists  of  three  essential  elements: 
independence,  verification,  and  validation: 

(1)  Independence.  An  organization/agency  which  is  separate  from  the  software 
development  activity  from  a  contractual  and  organizational  standpoint. 

(2)  Verification.  The  evaluation  to  determine  whether  the  products  of  each  step  of  the 
computer  program  development  process  fulfill  all  requirements  levied  by  the  previous  step. 

(3)  Validation.  The  integration,  testing  and/or  evaluation  activities  carried  out  at  the 
system/subsystem  level  to  evaluate  the  developed  computer  program  against  the  system 
specifications,  and  the  user's  and  supporter's  requirements  (AFR  800-14). 

Initial  Operational  Capability  (IOC) 

(JCS  PUBl) 

The  first  attainment  of  capability  to  employ  a  weapon,  item  of  equipment,  or  system  of  approved 
specific  characteristics  which  is  manned  or  operated  by  trained,  equipped,  and  supported  military 
unit  or  force. 

Integrated  Logistics  Support  (ILS) 

(AFR  800-8) 

ILS  is  a  disciplined,  unified,  and  iterative  approach  to  the  management  and  technical  activities 
necessary  to  (1)  integrate  support  considerations  into  system  and  equipment  design;  (2)  develop 
support  requirements  that  are  related  consistently  to  readiness  objectives,  to  design,  and  to  each 
other;  (3)  acquire  the  required  support;  and  (4)  provide  the  required  support  during  the  operational 
phase  at  a  minimum  cost. 

Interface  Control  Working  Group  (ICWG) 

(MIL-STD-483) 

For  programs  which  encompass  a  system/HWCI/CSCI  design  cycle,  an  ICWG  normally  is 
established  to  control  interface  activity  between  contractors  or  agencies,  including  resolution  of 
interface  problems  and  documentation  of  interface  agreements. 

Interoperability 


(AFOTECP  5) 
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A  measure  of  the  degree  to  which  computer  hardware/software  can  interface  to  and  operate  with 
other  similar  computer  hardware/software. 

Maintainability 

(AFOTECP  5) 

The  probability  that  a  system  out  of  service  for  maintenance  can  be  properly  repaired  and 
returned  to  service  in  a  stated  elapsed  time. 

Maintenance  (Software  Change)  Complexity 

(CURRENT) 

The  general  degree  of  difficulty  to  complete  a  software  change  (maintenance)  request:  high, 
medium,  low. 

High:  A  maintenance  action  where  changes  are  in  requirements,  design,  code,  and  test;  or 
greater  than  10  percent  of  a  CSCI  is  affected;  or  several  modules  are  affected  by  the  change  (global 
changes);  or  the  technical  nature  of  the  change  requires  highly  specialized  personnel  skills;  or  the 
level  of  effort  by  personnel  is  large. 

Medium:  A  maintenance  action  where  changes  are  in  design,  code,  and  test;  or  between  1 
percent  and  10  percent  of  a  CSCI  is  affected;  or  at  least  two  modules  are  affected  by  the  change 
(semilocal);  or  the  level  of  effort  by  personnel  is  average. 

Low:  A  maintenance  action  where  changes  are  isolated  to  only  one  unit  (e.g.,  one 

module/compilation  unit)  of  code;  or  no  more  than  1  percent  of  a  CSCI  is  affected;  or  the  level  of 
effort  by  personnel  is  minimal  (small). 

Maintenance  Documentation 

(AFOTECP  5) 

The  documentation  that  describes  the  maintenance  of  computer  system  hardware  and  software. 
Maintenance  (Software  Change)  Priority 
(CURRENT) 

The  criticality  of  the  software  change  (maintenance)  request  in  order  to  preserve  mission 
readiness:  emergency,  urgent,  and  normal. 

Emergency:  A  maintenance  action  requiring  all  available  personnels  dedicated  effort  to  correct 
the  problem  as  soon  as  possible  (e.g.,  24  hours);  mission  impact  -  mission  termination  or  severe 
degradation,  no  workaround. 

Urgent:  A  maintenance  action  requiring  next  "block  release"  turnaround;  mission  impact  - 
degraded,  requires  workaround. 

Normal:  A  maintenance  action  not  in  the  Emergency  or  Urgent  categories;  mission  impact  -  not 
degraded  but  inconveniences  occur. 

Maintenance  Profile 

(CURRENT) 

See  Historical  Maintenance  Profile. 


AFOTECPAM  99-102  Volume  2  Attachment  4  1  August  1994 


251 


Maintenance  (Software  Change)  Request  Category 
(CURRENT) 

The  identification  of  a  software  change  (maintenance)  request  by  specification  of  the 
maintenance  type,  priority,  and  complexity. 

Maintenance  (Software  Change)  Tjrpe 

(CURRENT) 

The  t3q)e  of  software  change  (maintenance)  action  required  to  complete  a  software  change 
(maintenance)  request:  conversion,  enhancement,  and  correction. 

Conversion  (Adaptation):  Any  change/effort  to  a  software  system  that  is  initiated  as  a  result  of 
changes  in  the  environment  (e.g.,  hardware,  system  software)  in  which  the  software  system  must 
operate. 

Enhancement  (Perfective):  Any  change,  insertion,  deletion,  modification,  or  extension  made  to  a 
software  system  to  meet  the  evolving  needs  of  the  user. 

Correction:  Any  change  that  is  necessitated  by  actual  faults  (induced  or  residual)  in  a  software 
system. 

Modularity 

(CURRENT) 

A  measure  of  the  extent  that  a  logical  partitioning  of  software  produces  into  parts,  components, 
and/or  modules  has  occurred. 

Mission  Critical  Computer  Resources  (MCCR) 

(AFOTECP  1) 

Computer  resources  incorporated  as  integral  parts  of,  dedicated  to,  required  for  direct  support 
of,  or  for  the  upgrading  or  modification  of  major  or  less  than  major  system(s).  Excludes  are 
automated  data  processing  (ADP)  resources  as  defined  and  administered  under  AFR  300  series 
(USAF/RD/  LE  Policy  Letter,  13  October  1981). 

Mission  Critical  Computer  System  (MCCS) 

(AFOTECP  1) 

(1)  A  computer  that  is  integral  to  an  electromechanical  system  and  that  has  the  following  key 
attributes: 

(a)  Physically  incorporated  into  a  large  system  whose  primary  function  is  not  data 
processing. 

(b)  Integral  to,  or  supportive  of,  a  larger  system  from  a  design,  procurement,  and 
operational  viewpoint. 

(c)  Inputs  include  target  data,  environmental  data,  command  and  control,  etc. 

(d)  Outputs  include  target  information,  flight  information,  control  signals,  etc. 
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(2)  In  general,  an  MCCS  is  developed,  acquired,  and  operated  under  decentralized 
management  (DoD  Directives  5000.1  and  5000.2). 

Operational  Support  Activity 

(CURRENT) 

Those  organizations  responsible  for  postdeployment  operation  and  support  of  a  system. 
Organizations  include  the  using  command,  supporting  command,  contractors  (if  used),  and  test  and 
evaluation  agencies  (if  used). 

Operational  Effectiveness 

(DoDD  5000.1) 

The  overall  degree  of  mission  accomplishment  of  a  system  used  by  representative  personnel  in 
the  context  of  the  organization,  doctrine,  tactics,  threat  (including  countermeasures  and  nuclear 
threats),  and  environment  in  the  planned  operational  employment  of  the  system. 

Operational  Suitability 

(DoDD  5000.1) 

The  degree  to  which  a  system  can  be  placed  satisfactorily  in  field  use  with  consideration  being 
given  to  availability,  compatibility,  transportability,  interoperability,  reliability,  wartime  usage 
rates,  maintainability,  safety,  human  factors,  manpower  supportability,  logistics,  supportability,  and 
training  requirements. 

Person-Months  per  Change  (PMPC) 

(CURRENT) 

Available  PMPC:  Raw  personnel  resources  workload  to  support  a  user/supporter  baseline 
workload  estimate  of  a  specified  number  of  changes.  Computed  as  the  number  of  full-time 
equivalent  personnel,  times  the  release  cycle  in  months,  divided  by  the  total  number  of  changes. 

Estimated  PMPC:  An  estimate  of  personnel  resources  workload  required  to  support  the 
user/supporter  baseline  estimate.  This  estimate  is  computed  by  using  a  regression  equation  whose 
coefficients  are  derived  from  historical  maintenance  release  data. 

Evaluated  PMPC:  A  realistic  estimate  of  personnel  resources  workload  effectiveness  to  support 
the  user/supporter  baseline  estimate  as  derived  from  an  evaluation  of  the  software  supportability 
characteristics . 

Personnel 

(CURRENT) 

See  Support  Personnel. 

Physical  Configuration  Audit  (PC A) 

(DoD  480A) 

The  formal  examination  of  the  "as-built"  configuration  of  a  unit  of  a  Cl  against  its  technical 
documentation  in  order  to  establish  the  CTs  initial  product  configuration  identification. 
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(ROWE) 

A  numerical  property  attached  to  an  activity  or  event  whereby  the  likelihood  of  its  future 
occurrence  is  expressed  or  clarified. 
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Probability  Distribution 

(ROWE) 

The  representation  of  a  repeatable  stochastic  process  by  a  function  satisfying  the  axioms  of 
probability  theory. 

Probability  of  Occurrence 

(ROWE) 

The  probability  that  a  particular  event  will  occur,  or  will  occur  in  a  given  interval. 

Procurement  Activity 
(CURRENT) 

Those  government  organizations  responsible  for  ensuring  delivery  of  a  production  system. 
Organizations  include  the  program  office,  implementing  command,  development  and  operational  test 
and  evaluation  agencies,  and  appropriate  independent  verification  and  validation  agencies  if  used. 

Product  Baseline 

(DoD  480A) 

See  Baseline 

Product  Configuration  Identification 

(DoD  480A) 

The  current  approved  or  conditionally  approved  technical  documentation  that  defines  the 
configuration  of  a  Cl  during  the  production,  operation,  maintenance,  and  logistics  support  phases  of 
its  life  cycle,  and  that  prescribes  (1)  all  necessary  physical  or  form,  fit,  and  function  characteristics  of 
a  Cl,  (2)  the  selected  functional  characteristics  designated  for  production  acceptance  testing,  and  (3) 
the  production  acceptance  tests. 

Program  Management  Directive  (PMD) 

(APR  800-14) 

The  official  HQ  USAF  management  directive  used  to  provide  direction  to  the  implementing  and 
participating  commands  and  satisfy  documentation  requirements.  It  will  be  used  during  the  entire 
acquisition  cycle  to  state  requirements  and  request  studies  as  well  as  initiate,  approve,  change, 
transition,  modify,  or  terminate  programs.  The  content  of  the  PMD,  including  the  required  HQ 
USAF  review  and  approval  actions,  is  tailored  to  the  needs  of  each  individual  program  (APR  800-2), 

Program  Management  Responsibility  Transfer  (PMRT) 

(APR  800-14) 

That  point  in  time  when  the  designated  Supporting  Command  accepts  program  management 
responsibilities  from  the  Implementing  Command.  This  includes  logistic  support  and  related 
engineering  and  procurement  responsibilities  (APR  800-4). 

Program  Support  Tools 

(AFOTECP  3) 

General  debug  aids,  test/retest  software,  trace  software/hardware  features,  use  of  compiler/link 
editor,  library  management/configuration  management/text  editor/display  software  tools. 
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Program  Test  Plan 

(AFOTECP  3) 

Set  of  descriptions  and  procedures  for  how  the  program  is  to  be  (or  can  be  or  has  been)  tested. 

Quality  Assurance  (QA) 

(CURRENT) 

All  actions  that  are  taken  to  ensure  that  a  development  organization  delivers  products  that  meet 
performance  requirements  and  adhere  to  standards  and  procedures. 

Release 

(CURRENT) 

A  version  of  a  software  system  representing  either  the  initial  baseline  configuration  or  an 
update  to  a  previous  version  that  incorporates  a  defined  set  of  software  change  requests.  Each 
release  becomes  a  new  baseline. 

Release  Engineering  Completion  Date 

(CURRENT) 

The  date  when  the  software  engineering  activity  for  a  release  is  complete.  The  software 
engineering  activity  includes  configuration  management,  quality  assurance,  and  software 
maintenance  project  phases  of  requirements,  design,  code,  unit  test,  integration  test,  and  operational 
test.  Activities  including  "kit  proofing,"  prom  burning,  and  in  general,  technical  order  modifications 
which  typically  occur  between  engineering  completion  and  field  implementation  (distribution)  are 
not  included. 

Release  Field  Date 

(CURRENT) 

The  date  when  a  software  system  release  is  officially  distributed  and  implemented  in  the  field 
for  operational  use. 

Release  ID 

(CURRENT) 

A  unique  identifier  for  a  software  system  release. 

Release  Start  Date 

(CURRENT) 

The  date  when  major  analysis  activity  related  to  a  specified  release  begins  for  which  software 
support  resources  are  required. 

Reliability 

(ROWE) 

The  probability  that  the  system  will  perform  its  required  functions  under  given  conditions  for  a 
specified  operating  time. 
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Requirements  Correlation  Matrix  (RCM) 

(AFR  57-1) 

A  management  tool  that  provides  a  single  source  of  traceability  and  comparison  of  the  user's 
system  requirements,  contractual  specifications,  and  operational  evaluation  criteria. 

Risk 

(ROWE) 

The  potential  for  realization  of  unwanted,  negative  consequences  of  an  event. 

Risk  Acceptance 

(ROWE) 

Willingness  of  an  individual,  group,  or  society  to  accept  a  specified  level  of  risk  to  obtain  some 
gain  or  benefit. 

Risk  Assessment 

(ROWE) 

The  total  process  of  quantifying  a  risk  and  finding  acceptable  level  of  that  risk  for  an  individual, 
group,  or  society.  It  involves  both  risk  determination  and  risk  evaluation. 

Risk  Assessment  Methodology  for  Software  Supportability  (RAMSS) 

(CURRENT) 

A  method  of  determining  the  disparity  between  the  estimated  risk  (determined  from  the  support 
concept,  baseline  software  supportability  profile,  and  historical  maintenance  profile)  and  evaluated 
risk  (determined  from  a  conversion  of  the  software  supportability  evaluation  metrics,  software 
product  maintainability,  software  support  resources,  and  software  support  life  cycle  processes). 

Simplicity 

(CURRENT) 

A  measure  of  the  extent  that  software  products  reflect  the  use  of  singularity  concepts  and 
fundamental  structures  in  organization,  language,  and  implementation  techniques. 

Simulation 

(AFR  800-14) 

The  representation  of  physical  systems  of  phenomena  by  computers,  models,  or  other 
equipment. 

Site 


(CURRENT) 

A  software  support  site,  of  particular  location,  where  software  support  activity  is  being 
accomplished.  Includes  sites  such  as  the  Air  Logistics  Centers  (ALC). 

Software 


(AFOTECP  1) 
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A  set  of  computer  programs,  procedures,  and  associated  documentation  concerned  with  the 
operation  of  a  data  processing  system. 

(CURRENT) 

The  programs  that  execute  in  a  computer.  The  data  input,  output,  and  controls  upon  which 
program  execution  depends  and  the  documentation  which  describes,  in  a  textual  medium, 
development  and  maintenance  of  the  program. 

Software  Change  Request 

(CURRENT) 

An  official  request  that  could  involve  a  change  to  a  software  system.  Such  requests  include 
problem  reports,  enhancement  requirements,  modification  requests,  or  any  other  forms  that  are 
officially  tracked  by  a  configuration  management  function. 

Software  Configuration  Management 

(CURRENT) 

A  discipline  applying  technical  and  administrative  direction  and  surveillance  to  (1)  identify  and 
document  the  functional  and  physical  characteristics  of  a  configuration  item,  (2)  control  changes  to 
those  characteristics,  and  (3)  record  and  report  change  processing  and  implementation  status. 

Software  Delivery 

(CURRENT) 

That  point  in  the  software  life  cycle  when  the  software  support  function  assumes  responsibility 
for  the  "next"  set  of  configuration  changes  to  the  software  (e.g.,  nest  block  release).  This  point  is 
logically  no  later  than  PMRT,  but  could  be  as  early  as  IOC.  This  applies  when  a  contractor  or 
government  agency  assumes  the  software  support  function. 

Software  Error 

(CURRENT) 

The  human  decision  (inadvertent  or  by  design)  that  results  in  the  inclusion  of  a  fault  in  a 
software  product. 

Software  Fault 

(CURRENT) 

The  presence  or  absence  of  that  part  of  a  software  product  that  can  result  in  software  failure. 

Software  Life  Cycle  Process 

(CURRENT) 

The  policy,  methodology,  procedures,  and  guidelines  applied  in  a  software  environment  to  the 
software  development  and  support  life  cycle  activities. 

Software  Maintainability 

(AFOTECP  1) 

The  ease  with  which  software  can  be  changed  in  order  to: 


(1)  Correct  errors. 
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(2)  Add/modify  system  capabilities  through  software  changes. 

(3)  Delete  features  from  programs. 

(4)  Modify  software  to  be  compatible  with  hardware  changes. 

(CURRENT) 

A  quality  of  software  that  reflects  the  effort  required  to  perform  software  maintenance  actions. 

Software  Maintenance 
(CURRENT) 

Those  actions  required  for: 

(1)  Correction  -  Removal,  correction  of  software  faults. 

(2)  Enhancement  -  Addition/deletion  of  features  from  the  software. 

(3)  Conversion  -  Modification  of  the  software  because  of  environment  (data,  hardware) 

changes. 

Software  Maintenance  Environment 
(CURRENT) 

An  integration  of  personnel,  support  systems,  and  physical  facilities  for  the  purpose  of 
maintaining  software  products. 

Software  Maintenance  Project  Management 

(CURRENT) 

The  software  life  cycle  process  management  applied  during  the  support  phase  for  the  software  to 
accomplish  specific  software  maintenance  tasks  that  derive  from  software  problem  reports  or  change 
requests. 

Software  Management 

(CURRENT) 

The  policy,  methodology,  procedures,  and  guidelines  applied  in  a  software  environment  to  the 
software  development/maintenance  activities.  Also,  those  personnel  with  software  management 
responsibilities. 

Software  Maturity 

(CURRENT) 

Software  maturity  is  a  measure  of  the  software's  evolution  toward  satisf5dng  all  documented 
user  requirements. 

Software  Project  Management 

(CURRENT) 

See  Software  Management. 
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Software  Project  Management  Design  Methods 

(CURRENT) 

The  software  project  management  process  utilizes  design  methods  that  enhance  software 
supportability  to  the  extent  that  design  methodology  standards  and  conventions  are  (1)  documented, 
followed,  and  validated  through  quality  assurance;  (2)  can  be  transitioned  to  the  support  activity; 
and  (3)  produce  adequate  design  specifications  that  reflect  supportability  characteristics. 

Software  Project  Management  Implementation  Methods 

(CURRENT) 

The  software  project  management  process  utilizes  implementation  methods  that  enhance 
software  supportability  to  the  extent  that  implementation/coding/testing  methodology,  standards, 
and  conventions  are  (1)  documented,  followed,  and  validated  through  quality  assurance;  (2)  can  be 
transitioned  to  the  support  activity;  and  (3)  produce  supportable  production  products. 

Software  Project  Management  Organizational  Structure 

(CURRENT) 

The  software  project  management  process  organizational  structure  enhances  software 
supportability  to  the  extent  that  the  physical  structure,  functional  responsibilities,  external 
interfaces,  and  assigned  personnel  provide  for  continuity  over  the  software  life  cycle  phases  and  have 
proper  interfaces  with  organizations  responsible  for  software  support. 

Software  Project  Management  Planning 

(CURRENT) 

The  software  project  management  process  utilizes  planning  that  enhances  software 
supportability  to  the  extent  that  plans  for  the  development,  test,  product  transfer,  and  operational 
support  exist  have  been  implemented,  have  been  appropriately  coordinated  across  activities,  and 
satisfy  contractual  and/or  regulation  requirements. 

Software  Project  Management  Project  Interfaces 

(CURRENT) 

The  software  project  management  possesses  organizational  interfaces  that  enhance  software 
supportability  to  the  extent  that  external  project  organizational  relationships  and  responsibilities  are 
(1)  defined,  (2)  provide  a  valuable  functional  role,  and  (3)  contribute  to  systematic  cost  effective 
procurement,  development,  and  operational  support  processes. 

Software  Project  Management  Test  Strategies 

(CURRENT) 

The  software  project  management  process  utilizes  strategies  that  enhance  software 
supportability  to  the  extent  that  the  test  plans,  descriptions,  procedures,  and  results  have  been  (1) 
documented,  (2)  can  be  transitioned  to  the  support  activity,  and  (3)  provide  for  a  consistent  and 
systematic  process  for  verifying  and  validating  that  software  requirements  have  been  satisfied. 

Software  Reliability 

(CURRENT) 

A  quality  of  software  that  reflects  the  probability  of  failure  free  operation  of  a  software 
component  or  system  in  a  specified  environment  for  specified  time. 
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Software  Portability 

(CURRENT)  A  quality  of  software  that  reflects  the  effort  required  to  transfer  the  software  from 
one  environment  (hardware  and  system  software)  to  another. 

Software  Support  Concepts 

(CURRENT) 

The  estimated  support  personnel  resources,  level  of  dedication  and  expertise  of  the  support 
personnel,  and  the  duration  of  the  block  release  cycle. 

Software  Support  Facility 

(AFOTECP  5) 

The  facility  which  houses  and  provides  services  for  the  support  systems  and  personnel  required 
to  maintain  the  software  for  a  specific  MCCS. 

Software  Support  Personnel 

(CURRENT) 

See  Support  Personnel. 

Software  Support  Resources 

(CURRENT) 

The  totality  of  personnel,  systems,  physical  facilities,  and  calendar  time  that  are  used/consumed 
during  a  software  release  effort. 

Software  Supportability 

(CURRENT) 

A  measure  of  the  adequacy  of  software  products,  resources,  and  procedures  to  facilitate: 

(1)  Modifying  and  installing  software. 

(2)  Establishing  an  operational  baseline. 

(3)  Meeting  user  requirements. 

Software  Supportability  Evaluation 

(CURRENT) 

An  evaluation  to  derive  a  measure  of  how  well  a  software  system  can  be  supported.  (See 
Software  Supportability.) 

Software  Supportability  Evaluation  Metrics 

(CURRENT) 

The  closed-form  questionnaire  scores  for  each  software  supportability  characteristic  in  a 
software  supportability  evaluation  as  well  as  the  values  computed  by  cumulating  lower  level  scores. 
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Software  Supportability  Negative  Outcome 

(CURRENT) 

Any  outcome  for  which  the  software  support  resources  are  not  adequate  to  accomplish  required 
software  support. 
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Software  Supportability  Risk 

(CURRENT) 

The  probability  at  a  given  point  during  the  software  support  phase  that  the  software 
maintenance  activity  specified  by  a  baseline  software  supportability  profile  cannot  be  accomplished 
with  the  available  software  support  resources. 

Estimated  Software  Supportability  Risk:  An  estimate  of  the  software  supportability  risk 
determined  by  the  area  under  a  normal  distribution  curve.  The  area  representing  risk  is  that  part 
under  the  curve  greater  than  the  subject  software’s  available  person-months  per  change  value  as 
computed  from  the  software  support  concept  and  baseline  software  change  profile.  The  normal 
distribution  curve  is  determined  by  using  the  estimated  person-months  per  change  as  the  mean  and 
the  standard  deviation  of  the  form  an  estimated  person-months  per  change  regression  equation.  (See 
Risk  Assessment  Methodology  for  Software  Supportability.) 

Acceptable  Software  Supportability  Risk:  The  estimated  software  supportability  risk  that  is 
agreed  upon  by  the  user  (using  command)  and  supporter  (supporting  command)  as  result  of  the 
baseline  software  supportability  agreement. 

Evaluated  Software  Supportability  Risk:  An  approximation  to  the  software  supportability  risk 
computed  from  the  software  supportability  evaluation  metrics.  The  computation  is  derived  from  a 
linear  regression  model  using  the  software  life  cycle  process,  software  product  maintainability, 
support  personnel,  support  systems,  and  support  facilities  as  the  five  regression  equation  factors. 

Measured  Software  Supportability  Risk:  See  Evaluated  Software  Supportability  Risk. 

Software  System 
(CURRENT) 

A  set  of  software  (specifications,  programs,  and  data)  that  constitutes  a  well-defined  major 
function  of  group  of  functions. 

Typical  Systems  include  avionics  operational  flight  programs  (OFF),  ground  based 
communications,  missile  guidance,  simulation,  threat  generators,  automatic  test  equipment  (ATE), 
and  electronic  warfare  (EW). 

Software  System  Type 

(CURRENT) 

One  of  seven  classifications  of  a  software  system's  primary  functional  mission:  ATD,  ATE,  C-E, 
EW,  OFF,  SIM,  and  SUF. 

ATD:  Aircrew  Training  Device  or  Operational  Flight  Trainer  (Weapon  System  Trainer)  for 
training  and  support  of  an  operational  system,  usually  in  the  form  of  a  replicated  cockpit  and 
instructor/operator  station. 

ATE:  Automatic  Test  Equipment  software  to  support  the  testing  of  hardware  units  under  test 
(UUT),  create  and  maintain  the  environment  where  the  test  software  may  be  used,  or 
prepare/analyze/maintain  test  software. 

C-E:  Communications-Electronics  software  for  command  and  control,  communications, 

surveillance  and  warning,  air  traffic  control,  intelligence,  and  other  related  functions. 
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EW:  Electronic  Warfare  software  that  involves  the  use  of  electromagnetic  energy  and  performs 
functions  either  separate  or  integral  to  a  larger  airborne  or  ground  system. 

OFF:  Operational  Flight  Program  software/firmware  that  is  integral  to  an  on-board  aircraft 
computer  system  including  navigation,  flight  control,  fire  control,  weapon  delivery,  electronic  engine 
control,  and  heads-up  display  (HUD). 

SIM:  Simulation  Software  not  included  as  part  of  the  ATD. 

SUP:  Support  Software  including  application  support  software  and  system  support  software  not 
included  in  any  other  category. 

Specification  Change  Notice  (SCN) 

(CURRENT) 

The  SCN  is  used  to  distribute  approved  page  changes  to  authorized  users  of  baseline  documents 
who,  in  turn,  are  responsible  for  posting  the  updates. 

Source  Code 

(CURRENT) 

The  form  of  the  program  code  in  its  source  language. 

Standards 

(AFOTECP  3) 

Procedures,  rules,  and  conventions  used  for  prescribing  disciplined  program  design  and 
implementation. 

Support  Concept 

(CURRENT) 

The  software  support  concept  usually  specified  as  part  of  the  CRLCMP.  Also  includes  that  part 
of  the  support  concept  necessary  to  establish  the  acceptable  risk  from  a  baseline  software  change 
profile:  standard  release  duration,  number  of  support  personnel,  average  skill  level,  percentage  of 
personnel  dedicated  to  releases,  support  facility,  etc. 

Support  Facility 

(CURRENT) 

The  physical  facility  resources  that  must  be  available  for  the  other  software  support  resources  to 
accomplish  a  specific  task. 

Support  Personnel 

(CURRENT) 

A  general  term  for  personnel  (military,  DoD  civilian,  or  DoD  contractor)  whose  skills  are 
necessary  to  directly  support  MCCS  software  maintenance.  Includes  but  is  not  limited  to 
management,  technical  and  nontechnical  support,  and  contractor  personnel. 

Support  System 


(AFOTECP  5) 
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Any  automated  system  used  to  change,  test,  or  manage  the  configuration  of  MCCS  software  and 
associated  documentation.  Includes  but  is  not  limited  to  host  processors,  software  bench  processors, 
laboratory-integrated  test  facilities,  operational-integrated  test  facilities,  and  automated  configura¬ 
tion  management  systems. 

Support  System  Facility 

(AFOTECP  5) 

See  Support  Facility. 

System  Software 

(AFOTECP  5) 

All  of  the  software  that  is  part  of  the  software  support  facility  computer  system.  It  is  never  or 
seldom  accessed  directly  by  software  support  personnel.  It  controls  the  processing  of  application 
software.  It  includes  operating  systems,  source  code  editors,  language  translators,  link  editors/load¬ 
ers,  librarian/file  managers,  data  base  managers,  and  automated  software  development  tools. 

Testability 

(AFOTECP  3) 

A  measure  of  the  extent  that  software  products  contain  aids  that  enhance  testing. 

Test  and  Evaluation  Master  Plan  (TEMP) 

(DoDD  5000.2-M) 

The  TEMP  is  the  basic  planning  document  for  all  test  and  evaluation  (T&E)  related  to  a 
particular  system  acquisition  and  is  used  by  the  Office  of  the  Secretary  of  Defense  (OSD)  and  all  DoD 
Components  in  planning,  reviewing,  and  approving  T&E.  The  TEMP  provides  the  basis  and 
authority  for  all  other  T&E  planning  documents. 

Traceability 

(AFOTECP  3) 

A  measure  of  the  extent  that  information  regarding  all  program  elements,  and  their 
implementation,  can  be  traced  between  all  levels  of  lesser  and  greater  detail  (e.g.,  up  or  down  in  the 
system  dociunentation  hierarchy). 

Uncertainty 

(ROWE) 

The  absence  of  information;  that  which  is  unknown. 

User/Supporter  Baseline  Estimate 
(CURRENT) 

A  set  of  conditions  agreed  upon  by  the  user  and  supporter  of  a  given  MCCS  that  represents  the 
best  estimate  of  the  expected  Baseline  Software  Change  Profile  for  the  first  (one  to  three)  releases 
after  Software  Delivery  and  the  planned  or  best  estimate  of  the  Software  Support  Concept. 
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VerificationA^alidation  (of  computer  programs) 

(AFR  800-14)  The  process  of  determining  that  the  computer  program  was  developed  in  ac¬ 
cordance  with  the  stated  specification  and  satisfactorily  performs,  in  the  mission  environment,  the 
functions  for  which  it  was  designed. 


